Linux 2.4 VM Documentation 115
popoutman writes "Mel Gorman has announced the availability of a guide to the 2.4 kernel VM
including a walkthrough of the VM code. Anyone interested in obtaining a solid understanding of the Linux 2.4 VM will certainly want to take a look at this documentation. Mel says that the effort is at least several weeks from being finished, but that he's releasing it now with the hopes of getting feedback to be sure he's on the right track.
He also notes that the 2.5 VM is still too much of a moving target for him to document it just yet." See also a Kerneltrap story.
Here's a tip for the author. (Score:2, Insightful)
Re:Here's a tip for the author. (Score:2)
Re:Here's a tip for the author. (Score:2, Interesting)
Re:Here's a tip for the author. (Score:1)
I assumed it was about Virtual Machines in user land as well.
Oh well, the fact that it's an incomplete document on something that noone but kernel hackers care about is irrelevent. It's only on
Re:Here's a tip for the author. (Score:3, Informative)
1. virtual memory (using secondary storage to allow the program to address more memory locations than there is in physical RAM)
2. virtual machine (a machine implemented in software on a real machine, which *could* have different opcodes, or the same ones as the real machine)
3. VM - the operating system by IBM which allows one of their mainframes to act like many (stands for virtual machine)
( Also, there's the VM in VMS, the DEC Vax and Alpha operating system, which stands for VIrtual Memory as in #1)
Best course feedback I ever heard (Score:2)
TLA=Three Letter Acronym, BTW
Re:Here's a tip for the author. (Score:1, Funny)
VM = Value Management
VM = Variable Metric
VM = Velocity Management
VM = Velocity Meter
VM = Velocity Modulation
VM = Version Manager
VM = VIBE Magazine
VM = Video Monitor
VM = Vietnam (Socialist Republic of Vietnam)
VM = View Mail (Emacs)
VM = Virtual Machine
VM = Virtual Madness
VM = Virtual Memory
VM = Vivamos Mejor (Let Us Live Better; Guatemalan agency)
VM = Voice Mail
VM = Voice Message
VM = Voice Modulation
VM = Voltmeter
but not that one!!!
Re:Here's a tip for the author. (Score:2, Informative)
As a (an ex-)Java guy I immediately thought it stood for "Virtual Machine."
Re:Here's a tip for the author. (Score:5, Insightful)
Better document... (Score:1, Flamebait)
Re:Better document... (Score:1)
Re:Better document... (Score:2, Informative)
Platforms supported:
acorn26 acorn32 algor alpha amiga amigappc arc arm32 atari bebox cats cesfic cobalt dreamcast evbarm evbmips evbppc evbsh3 evbsh5 hp300 hp700 hpcarm hpcmips hpcsh i386 luna68k mac68k macppc mipsco mmeye mvme68k mvmeppc netwinder news68k newsmips next68k ofppc pc532 playstation2 pmax pmppc prep sandpoint sbmips sgimips shark sparc sparc64 sun2 sun3 vax x68k x86_64
Re:Better document... (Score:2)
Also, since we're talking VM systems here, I see little need for a CPU with an MMU in a toaster and you'd likely be using the MMU-less Linux in that case. However there may be valid reasons for needing virtual memory in your toaster's OS but I really just want something that slightly burns bread (yes I know about the voice controlled one done at UW) which can be done very cheaply these days. Bettter be a fancy toaster!
Re:Better document... (Score:1)
JVM (Score:1, Interesting)
Kind of like FreeBSD did, fast JAVA under linux would be awesome.
Re:JVM (Score:5, Funny)
Re:JVM (Score:4, Funny)
Re:JVM (Score:5, Funny)
There used to be a kernel-space HTTP server. It was integrated into the kernel for a specific reason: zero-copy access to the network interface memory. It was fast and relatively feature-poor. If it crashed (fortunately, a rare occurance), you got a kernel panic.
Along came a user-space, zero-copy HTTP server. It was faster and had a few more features to boot. Being a user-space program, if it crashed, you got a core-dump. It could also be run in a chroot jail, a gigantic step more secure than running in ring-0.
Two lessons can be read from this:
1. Don't integrate something with the kernel unless there is a specific advantage you hope to gain from it. Will making a JVM part of the kernel really speed it up? Are you sure?
2. Don't under-estimate the speed of a properly designed user-space Linux program. The kernel developers have done a magnificant job tuning the kernel and providing APIs for performance-critical apps.
always nice to see some documentation (Score:2, Insightful)
i'm sure those who need to know though are full aware of the vm workings of the 2.6 kernel (ibm, redhat, oracle, rik, google, etc).
Linux: the big player everyone likes to root for, but noone wants to put in the lineup.
MORE documentation needed. (Score:1)
Re:MORE documentation needed. (Score:1)
here's the first one i found relevant. i don't know if it's the current implementation or not. i don't follow that stuff too much. i just save files and use whatever doesn't fsck every time the system crashes or locks up.
i like the method of releasing specs that implementations are to adhere to. and deviation from that should then be documented. W3C i believe was to accomplish a similar goal, but somehow they weren't too successfull.
Re:MORE documentation needed. (Score:2)
make sgmldocs psdocs pdfdocs htmldocs mandocs
However, I also subscribe to the relevant mailing lists, and review them daily.
HTH
Re:TROLL - Please mod parent down (Score:1)
-- Jack
Re:TROLL - Please mod parent down (Score:1)
See it here...
http://www-whois.internic.net/cgi/whois?
Re:This should be helpful (Score:5, Funny)
Define your function "Olog", please. Surely Mr. "Wagner LLC Consulting Co. - Getting it right the first time" couldn't have meant O(log n)... :)
Re:This should be helpful (Score:1)
Define your function "Olog", please. Surely Mr. "Wagner LLC Consulting Co. - Getting it right the first time" couldn't have meant O(log n)... :)
I believe the OP was in fact referring to the Olog-hai, the race of trolls enhanced by Sauron. HTH.
Re:This should be helpful (Score:2)
Re:This should be helpful (Score:1)
Mod Parent Down (Score:1)
Re:This should be helpful (Score:2)
Re:This should be helpful (Score:2)
Is late documentation a problem with Open Source? (Score:1)
Re:Is late documentation a problem with Open Sourc (Score:1)
The real problem is "release early (without documentation)" rule.
Re:Is late documentation a problem with Open Sourc (Score:2)
The cure that I use for this is to read the code comments and subscribe to the mailing lists.
Where I work (Score:5, Funny)
Well maybe not all the time and with serveral itterations, but I only manage your credit raiting, not you kernel VM.
every developer worth his salt knows... (Score:5, Funny)
Re:every developer worth his salt knows... (Score:2)
Re:every developer worth his salt knows... (Score:2, Funny)
Links to actual documentation (Score:1)
Here are the links for interested readers:
http://www.kerneltrap.com/node.php?id=5
Actual documentation:
The documentation comes in two parts. The first is "Understanding theLinux Virtual Memory Manager" and it does pretty much as described. It isavailable in three formats, PDF, HTML and plain text.Understanding the Linux Virtual Memory Manager
PDF: http://www.csn.ul.ie/~mel/projects/vm/guide/pdf/u
HTML: http://www.csn.ul.ie/~mel/projects/vm/guide/html/
Text: http://www.csn.ul.ie/~mel/projects/vm/guide/text/
PDF: http://www.csn.ul.ie/~mel/projects/vm/guide/pdf/c
HTML: http://www.csn.ul.ie/~mel/projects/vm/guide/html/
Text: http://www.csn.ul.ie/~mel/projects/vm/guide/text/
Re:Links to actual documentation (Score:2)
Or does he use a patched latex2html?
VM: Does it really matter? (Score:4, Interesting)
I'm not trying to troll, and maybe someone can explain this to me, but as far as I know, VM is used when a computer runs out of RAM. You take some of what's in the RAM and toss it on to the hard drive. When you need it again, you reload it back into RAM and use it.
I've got 1.5GB of RAM in the box I'm typing this on, and the only reason I don't have more is that I have no use for it.
At my work, we've got some systems that can eat up 2GB of RAM, but we just put 4GB in the box and don't worry about it. We've found that if we let anything touch VM it slows things down way too much.
Anyways, I was hoping someone could enlighten me as to why VM has been such a big deal recently.
Re:VM: Does it really matter? (Score:5, Informative)
This is a good question -- there are several features the VM system allows for in addition to overcommitting physical memory. If your system has enough memory to run with no swap/paging space set up, the VM system still provides these features without ever paging or swapping out memory.
The first is demand paging. That is, only those pages of an executable that are needed are brought into memory, and on-demand. Yes, this saves memory, but more importantly, it makes program startup much faster. Without demand paging, the whole program would need to be brought into memory at startup. Nowadays, when disks are much slower relative to CPUs, than they used to be, this makes a big difference, especially in the Unix shell-pipeline style of programming, where you run many different short-lived programs.
Secondly, the VM system allows for shared memory segments between processes. This allows for shared memory, threads and shared libraries.
Finally, the VM system implements caching of the file system, which we all know and love as a good use for all the memory we stuff into our machines now.
Re:VM: Does it really matter? (Score:2, Informative)
-- Jack
Re:VM: Does it really matter? (Score:5, Interesting)
Good point -- this is such a given on Unix systems that I didn't word it very well. What I should have said is that it enforces memory separation and protection between most memory segments and most processes, and allows for sharing of segments when explicitly setup. This is perhaps the most important thing the VM system does.
Re:VM: Does it really matter? (Score:3, Informative)
Another benefit is sparse allocation. When memory is allocated, it's all mapped to a zero filled page and marked copy on write. That allows programs to allocate massive arrays that are expected to be primarily filled with zeros without actually requiring that much memory. When a value is writen, a new page is allocated and mapped into the virtual space.
If desired, a file can be mmaped into the array instead to provide a simple persistant storage.
IIRC, suspend to disk is (or at least can be) also accomplished through VM.
The VM also allows an x86 system to have more than 4GB of ram. No single process can have more than a total of 4G (counting the kernel space) but using PAE, more than a total of 4GB can be allocated to multiple processes through the VM.
Although Linux doesn't do this, some OSes completely replace the filesystem with a VM. All files and directories are objects and are demand paged.
Re:VM: Does it really matter? (Score:3, Insightful)
Mmap is way, way more useful than that.
The key benefit of mmap is that on decent OSes (i.e. not NT/2000/XP), you get effectively get the buffer cache for the file mapped into your address space. Without mmap, you'd effectively have two copies of the file go through memory: one in the buffer cache and one in the application.
This is not so serious if your file is a sequentially-read stream, because the application won't hold much at a time. However, if it's a randomly accessed file with a large working set (e.g. in a database server), the win is huge.
Re:VM: Does it really matter? (Score:3, Informative)
Re:VM: Does it really matter? (Score:1)
VM does NOT mean just paging/swapping. that is a small part of it. mac os and windows users constantly misuse the term.
In all fairness, if under the old MacOS you booted with "virtual memory" off then virtual memory (in the correct sense) really was disabled.
Re:VM: Does it really matter? (Score:1)
Abstract:
Memory Management Units (MMUs) are traditionally used by operating systems to implement disk-paged virtual memory. Some operating systems allow user programs to specify the protection level (inaccessible, readonly, read-write) of pages, and allow user programs to handle protection violations, but these mechanisms are not always robust, efficient, or well-matched to the needs of applications. We survey several user-level algorithms that make use of page-protection techniques, and analyze their common characteristics, in an attempt to answer the question, "What virtual-memory primitives should the operating system provide to user processes, and how well do today's operating systems provide them?"
Re:VM: Does it really matter? (Score:1)
three basic reasons
1. resource virtualization
2. information isolation
3. fault isolation
Re:VM: Does it really matter? (Score:1)
Re:VM: Does it really matter? (Score:2, Informative)
Virtual Memory means virtualizing a process's access to memory such that the memory addresses it accesses isn't necessarily the same physical memory address. It allows for complete memory separation between processes, which offers a vast amount of protection so that processes can't crash other processes.
The only alternative to Virtual Memory is to use actual physical memory addresses. This was exactly the case with DOS. In DOS you would have to manage memory and make sure your process (it's single-threaded... remember?!?) didn't overwrite memory that it didn't own. Everything in memory could be accessed from every single process.
With virtual memory, two processes could access memory location 0x1000000 and it would map to different physical memory addresses. In fact, the OS will give each process a full memory address range that is completely usuable by the process without any fear of stepping over someone else's memory. This effectively separates processes and prevents a hell of a lot of corruption because of memory bugs in programs.
This really simplifies programming because whenever we get a memory address from malloc, we don't have to do anything about making sure we don't step over someone else's memory information.
What you are in fact talking about is paging, the act of having a backing store of information on hard drive. This allows you to keep track of more memory than you physically have, by swapping memory back and forth from physical memory to a hard drive. Any unused memory would be paged to disk.
Yes, today you can have gigabytes of memory, and when running Mozilla, boy you really don't need virtual memory.
But what about 10 years from now, when Microsoft Office takes up 50 GB of hard drive space, and 16 GB of memory?
Virtual memory is a way of abstracting memory use on a per process basis, and it's inclusion in most modern operating systems is one of the reasons why things are relatively stable these days, compared to the old DOS days, anyway.
Re:VM: Does it really matter? (Score:5, Informative)
Because it's important. :-)
Some people think that virtual memory means paging to secondary memory, such as disk. It doesn't. Virtual memory systems can support this, but many OSes (e.g. QNX) support virtual memory with no disk paging. (OK, QNX does support disk paging, but only as an afterthought, so that QNX can be self-compiled. GCC takes a lot of memory.)
Virtual memory provides a virtual address space for each process. The benefits include:
This is just off the top of my head.
Re:VM: Does it really matter? (Score:1)
*_g_o_a_t_s_e_x_*_g_o_a_t_s_e_x_*_g_o_a_t_s_e_x_*
g_______________________________________________g
o_/_____\_____________\____________/____\_______o
a|_______|_____________\__________|______|______a
t|_______`._____________|_________|_______:_____t
s`________|_____________|________\|_______|_____s
e_\_______|_/_______/__\\\___--___\\_______:____e
x__\______\/____--~~__________~--__|_\_____|____x
*___\______\_-~____________________~-_\____|____*
g____\______\_________.--------.______\|___|____g
o______\_____\______//_________(_(__>__\___|____o
a_______\___.__C____)_________(_(____>__|__/____a
t_______/\_|___C_____)/______\_(_____>__|_/_____t
s______/_/\|___C_____)___ISA_|__(___>___/__\____s
e_____|___(____C_____)\______/__//__/_/_____\___e
x_____|____\__|_____\\_________//_(__/_______|__x
*____|_\____\____)___`----___--'_____________|__*
g____|__\______________\_______/____________/_|_g
o___|______________/____|_____|__\____________|_o
a___|_____________|____/_______\__\___________|_a
t___|__________/_/____|_________|__\___________|t
s___|_________/_/______\__/\___/____|__________|s
e__|_________/_/________|____|_______|_________|e
x__|__________|_________|____|_______|_________|x
FIGURE 1
Why this has never been fixed is a mystery to me.
GSA PUBLIC KEY
asl;dfhlaskhdflkhasdflkasdhkflhasldfhjaskhdk
Re:VM: Does it really matter? (Score:2)
Because VM is not just used for swap. It's also used for memory protection, buffer caching, shared libraries, demand paging, and dozens of other performance improvements.
Re:VM: Does it really matter? (Score:1)
Documentation and uncommunicative maintainer (Score:4, Informative)
Re:Documentation and uncommunicative maintainer (Score:2)
Ok, I sympathise with his daily email overload but maybe he should delegate some of his workw , like dealing with kernel email? There are many other qualified people who have offered to help with maintaining 2.4.x kernels. That would be better than silently dropping email. Maintaining a relatively common Linux kernel series like 2.4.x is such an important job we should not have to rely on one overloaded person -- that's an unnecessary bottleneck. This is not a criticism of Marcelo's quality of work, which I think is very high. I am suggesting that he should delegate at least the work of dealing with kernel email.
Re:Documentation and uncommunicative maintainer (Score:2)
From other people's comments it sounds like Marcelo is too overloaded even to deal with all the bug fixes he receives by email, let alone the bug reports. He silently drops fixes as well as bug reports, so my point is he should delegate some of his heavy workload, such as dealing with email, to the many other qualified people who have offered to help as maintainers of 2.4.x kernels. For some reason he hasn't accepted any of those offers of help.
LKML is a very high traffic mailing list. I'm wary of adding to that flood of email. However, due to the lack of response from Marcelo I will probably email LKML.
Different VM (Score:4, Interesting)
Here's what I'm thinking. Ditch pages and memory contexts entirely. Instead, divy up a 64-bit virtual address space among individual processes, say 48 bits apiece. If a process wants to access memory outside it's 48-bits, it would need to access it through special pointers (which, thanks to a tagged-RAM architecture) could written to by the OS (allowing the OS to define its own protection and sharing mechanisms). Does anybody know of any existing systems that work even vaguely like this? Or of a different MMU architecture at all? I was hoping that AMD might at least include software TLB management, because there is some nifty stuff you can do with that, but it looks like Hammer will use the same VM mechanism that came out with the 386!
Re:Different VM (Score:2)
Look, it's AS/400!
Doing something "just for the hell of it being new" doesn't seem too popular among CPU manufacturers who are trying to actually make money.
Re:Different VM (Score:2)
PS> Hm. I knew AS/400 had a tagged RAM architecture, but I was under the impression that it was for marking crypto data. Could a scheme like the above be implemented on AS/400, and if so, where can I find some reference info about the MMU?
Re:Different VM (Score:2)
I knew AS/400 had a tagged RAM architecture, but I was under the impression that it was for marking crypto data. Could a scheme like the above be implemented on AS/400, and if so, where can I find some reference info about the MMU?
My impression is that OS/400 has a single address space and a single level store. The hardware is mostly undocumented AFAIK.
Re:Different VM (Score:1)
Maybe, but in my opinion people who just like to try different stuff (because it might be better, or because it interests them to try it, or just because...) tend to be the types who eventually come up with something better. While people who don't want to try anything because they worry they might be "reinventing the wheel" or that "maybe it won't be better" or whatever....probably are unlikely to ever create something useful.
Usually the best ideas are not arrived at directly, but after exploring many different ideas along the way. Most of those ideas are probably not better than what they replace, but they might be a stepping stone to a really great idea.
(sorry to go further off topic, but I just hate to see creativity squelched
Re:Different VM (Score:2)
Windows CE 2 (Score:1)
copulating modules (Score:2, Funny)
Generally, related operations are clustered together in the same file and there is not much copulation between modules.
I hope the discovery channel will do a special on copulating modules some day. I've seen module coupling but no copulation yet.
Re:copulating modules (Score:2, Funny)
> modules some day. I've seen module coupling but no copulation yet
Haven't you ever heard of sexy code? Is is meant to be taken literally, isn't it?
(typo corrected in the work in progress version)
So any clues why VM goes zombie? (Score:3, Interesting)
My kswapd went zombie after ~18days uptime; it had gone zombie after about 70 days prior to that reboot. I've found references to this occurance
Fortunately, I have plenty ram and this doesn't seem to otherwise affect anything.
Re:So any clues why VM goes zombie? (Score:2)
I got the feeling this was always from bad RAM/motherboard etc.
Re:Pretty good English for a foreigner (Score:1)
It's
Re:Pretty good English for a foreigner (Score:2)
Re:Pretty good English for a foreigner (Score:1)
Re:Pretty good English for a foreigner (Score:1)
Re:Pretty good English for a foreigner (Score:1, Offtopic)
Here you go fellows, an excellent example of the [hopefully minority] view that America is the center of the universe.