How To Use a Terabyte of RAM

Spuddly writes with links to Daniel Philips and his work on the Ramback patch, and an analysis of it by Jonathan Corbet up on LWN. The experimental new design for Linux's virtual memory system would turn a large amount of system RAM into a fast RAM disk with automatic sync to magnetic media. We haven't yet reached a point where systems, even high-end boxes, come with a terabyte of installed memory, but perhaps it's not too soon to start thinking about how to handle that much memory.

Add-Free one-page Version of the story

[saibot834]

For those of you who don't have Adblock: Printerfriendly Version [idg.com.au]

Power Failure

[Anonymous Coward]

One important thing to consider, is that if using a ramdisk for important stuff, what happens when the power dies?

For example, will the stuff synced from magnetic media be stored elsewhere? If so, what happens to the speed?

-B

uh - there is at least one system with 1TB of RAM

[Anonymous Coward]

You wrote: "We haven't yet reached a point where systems, even high-end boxes, come with a terabyte of installed memory" - this is not true. Sun's E25k can go over 1TB of memory.....

nothing new here

[dltaylor]

Linux gobbles free RAM to add to the buffer cache. This is already a large RAM disk with automatic sync. In embedded systems, you can even decouple the buffer cache from any physical media and just live in a variable size RAM disk, which means that Linux finally catching up to AmigaDOS.

Re:1 TB of memory...

[Digi-John]

emacs is a Lisp interpreter, an editor, a games package, an irc client, many things, but its memory usage is just a drop in the bucket compared to the monstrosities I mentioned above. Of course, there's quite a few complete operating systems that can boot in the amount of RAM required by emacs :)

Re:Power Failure

[itsjz]

There's about three paragraphs in the article discussing this. Basically, use a UPS:

If line power goes out while ramback is running, the UPS kicks in and a power management script switches the driver from writeback to writethrough mode. Ramback proceeds to save all remaining dirty data while forcing each new application write through to backing store immediately.

Re:The problem with giving Windows 1TB...

[W2k]

If you run MS SQL Server and don't manage the RAM then it will use it all just for the fun of it.

If you find this in any way strange, wrong or confusing, perhaps you should read up as to what the primary purpose of a frikkin' DATABASE SERVER is.

Here's a hint: the more data it can keep readily accessible (that is, in RAM) the better it will perform. And as you mentiones, you can of course set it to use less RAM if you have to. It's just that it's optimized for performance by default.

Re:You only need 16GB of RAM for this to be useful

[exley]

Things like this [sourceforge.net] (somewhat smaller scale) already are [gentoo.org] (somewhat bigger scale) being done.

Re:Add-Free one-page Version of the story

[Freedom Bug]

Here's a much better link on Jon Corbet's own site, the famous Linux Weekly News:

http://lwn.net/Articles/272011/ [lwn.net]

Re:1TB of RAM is available today in a server

[thinduke]

IBM p595 [ibm.com] can have 1TB of RAM too. And yes, they run Linux.

Re:1TB of RAM is available today in a server

[thatskinnyguy]

Something like this one [sun.com]?

Re:Add-Free one-page Version of the story

[Anonymous Coward]

Or you could just read the story on LWN [lwn.net].

Out of curiosity, why the link to PC World when the summary specifically mentions that it's on LWN? If Jon get paid for the PC World version because of the ad revenue, that's fine with me. However saying that it's an LWN piece and linking to another source is a bit misrepresentative.

cachefs

[argent]

A fully caching file system that could be layered on top of your network or disk file system. Sun did this for dataless workstations and it worked pretty well.

Another historically interesting ram file system was the Amiga Recoverable RAM Disk. You coudl even boot off it.

Re:uh - there is at least one system with 1TB of R

[Dammital]

IBM's just-announced z10 mainframe, with 1.5 TB memory [ibm.com].

Re:8 GB

[Gr8Apes]

I run Oracle on Linux - it barely fits into 4GB. Add in a few other daemons, and I can easily fill 8GB.

Re:You only need 16GB of RAM for this to be useful

[Daniel Phillips]

Yeah, imagine, then, to be able to use such a fast disk as your swap device! That'll make your system swiftz0rs.

Bingo. That is one way you can use the Violin 1010 [violin-memory.com] without needing any special backing to disk at all. In fact, this is a nigh-on perfect use of the device, because the 2x8x PCI-e bus connection, while fast, is still not as fast as main memory. But the swap subsystem knows now to manage that latency increase quite nicely. Such a swap arrangement will even tend to bring things back in balance as far as the Linux VM goes, since in the good old days when swap was invented, disk was only two or three orders of magnitude slower than memory instead of 5 orders like today.

Or, hey, wait a minute... In all honesty, though, I don't really get the point of this. Isn't the buffer cache already supposed to be doing kind of the same thing, only with a less strict mapping?

Indeed, less strict, and it also does not know how to flush dirty cache to disk and switch to synchronous mode when running on UPS power, or how to fully populate the cache as fast as possible on startup. Indeed, buffer cache can be taught these things, but I have already created a block driver to do it, which has the added benefit of supplying a nice general interface that will no doubt be repurposed in ways I did not think of. Maybe after Ramback is really solid I will create a variation that sits right in the VFS, though actually there are other more important projects in the pipe so anybody who wants to do that, be my guest.

Re:You only need 16GB of RAM for this to be useful

[shapor]

posix_fadvise() technically does allow you to do what you want. You can use posix_fadvise(POSIX_FADV_DONTNEED) to evict the buffer cache in the IO loop of the program. See http://insights.oetiker.ch/linux/fadvise.html [oetiker.ch] for the ugly details. Unfortunately you can't just make one system call and have it effect an entire file or process. POSIX_FADV_NOREUSE is supposed to be the default in the kernel buffer cache management so it is implemented as a no-op.

Re:1 TB of memory...

[Anonymous Coward]

It does indeed, the implementation of vi for emacs is called viper, there might be another one as well, but I'm not sure.

cosmology simulations

[Anonymous Coward]

Speaking of voxels, octrees, and a terabyte (TiB that is; 2^40 bytes):

Cosmology simulations take as much memory as you can throw at them. You think a few exabytes is big, but that's just peanuts to space!

This one is one is the most complex (by most measures) cosmological simulation ever run, using just over 10 billion particles and nearly the entire terabyte of the computer's available memory:
http://www.mpa-garching.mpg.de/galform/millennium/ [mpa-garching.mpg.de]

They ran it on a 512-CPU IBM p690, which took about 28 days of parallel runtime. (2^40) / (10^10) ~= 100 bytes per particle, which is high by maybe 50% but accounts for overhead and is a reasonable value. The paper reports ~200 Gflops sustained, which is certainly memory bound in any distributed N-body simulation, giving ~5*10^17 total floating-point ops.

They only did total output at 64 output timesteps because each of those took about 300GB for ~20TB total, which filled up a decent sized storage rack 3+ years ago (and still). The contemporary highest-detail galaxy clusters were modeled with as many as 3 million particles, but the simplest interesting "structures" (the dark-matter halo detection threshold was 20 particles) contained 20~100 particles.

Now for the cool stuff:
- there are ~400 billion galaxies in the observable universe, even if galaxy surveys (Sloan, etc.) catalog only a small fraction of them
- reasonable (i.e. "useful") resolution for typical structures of interest (quasars, typically) is ~1~10 million particles; better if you can get it

So:
You have to model dark matter halos with useful precision. Even if you are satisfied with modeling whole galaxies as point-like objects:
(4*10^11 galaxies) * (100 bytes/particle) = 4*10^13 bytes ~= 40 TB
(~10^12 CDM particles) * (100 bytes/particle) = ~100TB
=> allowing a little slush, 200TB of RAM would get you some very valuable theoretical simulation data of an object/construct/whatever that approaches literally the size and complexity of the observable universe at a galactic scale. More memory will just make that even better, too.

Desktop memory doubles say, every 2 years; look at the typical "consumer" systems the last few years: ...
1994: 16MB (P1... wow; 16 is a little high for '94 I guess)
1996: 32MB
1998: 64MB
2000: 128MB
2002: 256MB
2004: 512MB
2006: 1GB
2008: 2GB

We want to see when we have about 100,000 times as much memory as now, which is about 16.5 doublings, each of which take around 2 years, so we should see this some time around 2035-2040 on consumer grade machines. A purpose-built supercomputer can contain, say, 1000 times the consumer amount, so today's 4~50TB supercomputers need to double, drumroll.... 2~5.5 times. The top supercomputers will probably have 200 terabytes of ram within 2 years, and the lowly supercomputers cosmology simulations have to settle for will probably have that within 10-12 years; i.e. by 2020 (yeah, it's getting closer!).

It may even be possible to do quad-precision floats in hardware, which would be beneficial for cosmological simulation (or any N-body simulation), and that will cost a doubling too. And after a petabyte or so, the resolution gets *really* interesting because you can start modeling macro-scale systems of micro-scale (you know, individual atoms or stars) as long as you know the force laws, and you can start finding those out numerically if you don't. So there's plenty of use for *truly* astronomical, dizzying amounts of memory way, way beyond one measly terabyte.