OpenMosix 146
Francesco Taurino writes "Moshe Bar has released a new Mosix system: openMosix.
From the site:
"For thousands of users Mosix has been a reliable, fast and cost efficient clustering platform. There are hundreds of Mosix installations in life sciences, finance, industry, high tech, research and government environments. The goal of openMosix is to give to these users a continued support and an up-to-date platform. openMosix is initially fully compatible with the last Mosix (1.5.2 for 2.4.13) kernel, but is now growing in its own direction.
If you would like to contribute to the openMosix project, drop a line to moshe@openmosix.org.""
OpenVMS and clustering (Score:3, Interesting)
Re:OpenVMS and clustering (Score:2, Funny)
Oh well, so much for downloading that and installing it on a PC for obscurity value.
Re:OpenVMS and clustering (Score:1)
Re:OpenVMS and clustering (Score:1)
Please mod the uninformed troll down... (Score:1, Offtopic)
Simple question: Why the split? (Score:2)
Only Prof. Amnon Barak is authorized to represent MOSIX.
...???
What's the story here, Moshe?
Re:Simple question: Why the split? (Score:4, Interesting)
After a difference of opinions on the commercial future of Mosix, I have started a new clustering company - Qlusters, Inc. - and Prof. Barak has decided not to participate for the moment in this venture (although he did seriously consider joining) and held long running negotiations with investors. It appears that Mosix is not any longer supported openly as a GPL project. Because there is a significant user base out there (about 1000 installations world-wide), I have decided to continue the development and support of the Mosix project under a new name, openMosix under the full GPL2 license. Whatever code in openMosix comes from the old Mosix project is Copyright 2002 by Amnon Bark. All the new code is copyright 2002 by Moshe Bar.
It has to happen. (Score:2, Funny)
Oh yeah, this is mosix, bummer.
Re:It has to happen. (Score:2)
So feel free to imagine what you like.
This is great... I think (Score:5, Interesting)
I'm not clear about some things though... How is MOSIX currently licensed? Why are they being so closed about development?
Re:This is great... I think (Score:2, Interesting)
License
mosix on freshmeat [freshmeat.net]
Re:This is great... I think (Score:2)
1.5.2 isn't latest MOSIX (Score:3, Informative)
Re:1.5.2 isn't latest MOSIX (Score:2)
Building a mosix cluster (Score:1)
Anyone have some good links for a howto? Or anyone want to share their experience?
Re:Building a mosix cluster (Score:4, Informative)
Unpack kernel sources.
Run the Mosix install script.
Did that on each node, then started the mosix service on each.
It worked like a charm for large computations, but had three flaws for normal use.
1) By default, it does not auto-migrate, which was pretty dumb. And getting it to auto-migrate was buried deep in the docs, though it could be guessed from reading up on locks. (echo 1 >
2) Migration only occurrs after a certain load average is maintained... if your job involves spawning multiple short-lived processes, like a large compile, it doesn't migrate anyway.
3) Network usage for migration was very heavy over Fast Ethernet.
There you have it. It's the last reason that MOSIX isn't used often in commercial clusters, but it seems well-suited for other distributed computing applications, and has some interesting features, especially for NOW configurations.
Re:Building a mosix cluster (Score:2)
Hmmm, maybe that is why it all of a sudden started working when I re-installed it. Anyway, I highly recomend MosixView [www.mosixview.com] for Mosix Administration. It is a effective but simple way to monitor and adjust your cluster.
2) Migration only occurrs after a certain load average is maintained
I believe that is what Prof Amnon is using for developing U-Mosix. From the home www.mosix.org page...
"U-MOSIX provides even load distribution using several of the algorithms of K-MOSIX. U-MOSIX is better tuned for cluster and GRID computing, including the ability to handle large number of short processes, run in heterogeneous clusters, with different versions of Unix such as FreeBSD, Linux and Solaris."
For those of us that don't want to wait for U-Mosix for grid-computing (also known as cluster queueing) I suggest Sun's Open Sourced Grid Ware Engine [sun.com]. It comes complete with a Beowulf Cluster built in.
3) Network usage for migration was very heavy over Fast Ethernet.
Actually, we haven't noticed much of a load at all.
Btw, we are a commercial cluster
Re:Building a mosix cluster (Score:1)
http://www-106.ibm.com/developerworks/linux/
Qlusters (Score:1)
Re:Qlusters (Score:1, Funny)
You are probably thinking of that cereal commerical where a squirrel is chasing this guy trying to get his "cluster of oats" cereal.
Re:Qlusters (Score:2)
I never heard of mosix, (Score:4, Informative)
Clusters kick ass! (Score:1)
I hope.. (Score:2, Funny)
.. that the Mosix site is running on a Mosix cluster to withstand the slashdot affect
Why the fork... (Score:4, Interesting)
Re:Why the fork... (Score:5, Interesting)
In the mosix website they speak about going to user-space, which is probably being done to avoid the GPL.
Just ask yourself, if they had Mosix for 2.4.17 (they released it soon after my openMosix, so it must have been there all along) ready, why didn't Prof. Barak give it to the user-community?
That's what I mean they are going proprietary.
Moshe Bar
Re:Why the fork... (Score:3)
Ok...that's not a great sign..
He complained to me repeatedly about the GPL
There are other licenses out there. Prof. Barak himself I suppose would ahve to answer what his issues with the GPL are. I wrote to him a while back and got the impression that he was not interested in the GPL
that Linus is a fool
If that's a quote, I've really lost respect for him. Linus is certainly not a fool.
In the mosix website they speak about going to user-space, which is probably being done to avoid the GPL.
While this is a possible outcome, this is a bit of fear-mongering...
Just ask yourself, if they had Mosix for 2.4.17 (they released it soon after my openMosix, so it must have been there all along) ready, why didn't Prof. Barak give it to the user-community?
actually, my first thought was that whomever hadn't gotten it to work with the new VM code, not that he was withholding it.
I guess my thought it is...He has the right to, I suppose. I'm glad that someone is taking it and keeping it not proprietary. It has the potential in some areas that I work with.
Re:Why the fork... (Score:1)
nuFF sed!
Re:Why the fork... (Score:1)
Why Not Fork?? (Score:1)
I for one, hope that the OpenMosix project moves forward, and makes quick and timely releases. I would like to see an openMosix for 2.4.17 and the upcoming 2.4.18! Great work guys!
Re:Why the fork... (Score:1)
I attended a lecture by Prof. Barak in Edinburgh sometime ago (so this next bit is fuzzy), but this is as I understand it;
Mosix started as a means of building high performance computing facilities, in the face of the export ban of HPC machines out of the US. It wasn't built to provide an excellent open clustering system, but to provide HPC facilites to University of (? Lebanon ?). I wouldn't be surprised if the military were involved. But certainly, it's been the University that's been paying the bills as it were, and so they'll have some (considerable, I suspect) say in what's going on.
Professor Barak struck me as a man very devoted to his work (he did give us a lecture in Edinburgh, after all), so don't diss him too readily.
This was all a while ago mind you, things may well have changed (drastically) since then.
Re:Why the fork... (Score:1)
Can't look inside his head from a distance, but calling someone a fool is not something that pops up in ones mind. After you announced the fork, maybe his bitterness was over come by his pride about Mosix...
ohwell, just my 2 cents. Hope that the Prof now becomes resonable, joining forces would be more productive than working seperated...
Re:I hope Mosix does away with HZ (Score:1)
Moshe Bar
Re:I hope Mosix does away with HZ (Score:2, Interesting)
It was a very bad decision to ever make a private kernel constant public (like HZ). There should have been a system function for it (or a
There was a thread on the lkml in the summer of 2001 where they discussed getting rid of HZ altogether and keeping the outward appearance of a phony HZ for just such broken userland programs. Why not take that approach?
HZ = 100 in this age of multi-GHz x86 machines is insane.
how does mosix deal with dead cluster members? (Score:2)
Re:how does mosix deal with dead cluster members? (Score:2)
I'm not sure, it just seems reasonable.
Re:how does mosix deal with dead cluster members? (Score:2, Informative)
However, if a node dies abruptly, the job may be lost (I'm not sure here, because it hasn't happened to me yet). Logically, the submitting node should see that the node has died and should re-submit to a good node. Anyone have a difinitive answer on this?
A variety of things, and your milage may vary (Score:3, Interesting)
We run Mosix, and have had jobs fail. If a node goes down that is running a process two things can (and have) happened. The process dies, or it restarts on another node. What determins this? I have no idea. I can only speak from observation.
It is suggested in the documentation that you have a large swap space on your disk to handle nodes going down. Perhaps with a cached copy of a process it will live on.
In any case, its not scientific but I thought I'd throw that in.
Admin FAQ (Score:2)
Re:Admin FAQ (Score:1)
What MOSIX can't do (Score:3, Informative)
There are several limitations to what MOSIX can
currently do. Java native-threads is one thing,
since MOSIX cannot migrate apps using shared
memory all native-threads applications will stay
on the node they started on, green-thread
based application can migrate though the
internal threads aren't exposed to the OS so no
real parallelism is achieved. MOSIX also can't
migrate sockets so I/O bound problems also
stay at home. Mixed I/O and CPU jobs can
migrate for CPU cycles, but are brought back for
the I/O ops. In the limited testing to date,
processes that can't take advantage of MOSIX
don't seem to be hindered at all by it
Re:What MOSIX can't do (Score:3, Informative)
with mosix, you just 'fork and forget'. that's simple enough. but here are some of the issues that i've run into and hope it's useful for someone else since it took me about a whole week to figure it out.
Sockets don't migrate. fileHandles dont' migrate. ie. for any I/O to occure, the process is brought back home, then upon completion of the I/O, if appropirate it is transparently sent out. what that means is that if the ratio of i/o to other normal crunching stuff is low, the process is probably going to spend most of it's time at home. what i'm going to do next is test if a process migrates at all by just having an open file handle without doing any I/O. i might post it as a follow-up to this comment. if i have time, i'll also play around with mfs. but probably not. if someone has more info regarding I/O using mfs i would appriciate much. ie. if mfs is used to do file I/O can the process still migrate while doing the I/O? apart from file sharing, what other benefits does mfs bring?
Another thing i've tested is java. as you might already know, native-thread java uses shared memory so that can't be migrated. i was using sun's VM so i turned on the '-classic' switch to use green threads. it migrates. for 5 instances of the JVM, on (3) pentium MMX boxes, i had a 70% (roughly) performance gain (timewise). for about 20 instances of the JVM (i know, that's a waste but more on that later..) i had about 47% gain in time. and i assume it will reduce drastically after that for just spawning that many JVM vs. doing actual work.
to optimize, what i'm going to do is first figure out how many nodes are running. then spawn that many JVMs and split the job between those VMs. and hope mosix sends out each VM to a seperate machine. this is not much different from running a rsh call (or ssh for the security minded folks) except, we don't have to figure out the load on each box we run the command on and we don't have to worry about collecting the output at the end of the run. if someone has a better idea about running java on mosix, i'd love to hear about it. just post your ideas as a reply to this message. i'm sure others will find it useful too. with the mailing list going away and no real documentation for mosix... bummer. also, has anyone tested it with apache? i don't know much about apache but know that it uses shared mem. so can the process forks of apache be migrated? + there's the issue of open sockets from a specific IP. how does that play into this. i've read stuff on web sites that say they're running apache on a mosix cluster. is it one of those 'ahh... i want a beowulf of these' nuts? or is that legit?
MOSIX question (Score:1)
If not, I may just have to try this one
Re:MOSIX question (Score:1)
This is not to say that you can't run PVM or MPI on top of MOSIX. In fact, IIRC, MPI actually runs BETTER when used in conjunction with MOSIX (???) Anyone have any hard numbers here?
Re:MOSIX question (Score:1)
then again, i might have read it in the mosix book (details on their homepage) or heard it in one of prof. barak's lecture.
Re:MOSIX question (Score:3, Informative)
From my brief experimentation with Mosix and a bit of reading, this sounds correct.
Basically, mosix is a very "chunky" sort of clustering - it works on the level of "whole processes". Because of this, you don't need to write your software to do the splitting and migrating yourself as you do with "less chunky" pvm and mpi. On the other hand, a process split off from a pvm program running can be handled by mosix like any other process and migrated to the cpu that mosix thinks can get the process finished fastest.
Mosix seems like an ideal way to 'lend' processing power to slower machines. This is what I was doing when I played with it previously - I had a K6/2-350 and a P-100 laptop with no L2 cache. I got Mosix set up on them both and used a command-line mp3 encoder as a benchmark. On the P-100, encoding speed was about 15% of realtime. On the K6/2, it was about 110%. Running Mosix between the two over 10Mb Ethernet, I could encode mp3 at about 85% - I suspect it'd have been significantly closer to 100% if I'd had 100Mb Ethernet at the time...
Hopefully OpenMosix will keep up with current kernel versions better. Better still, maybe they'll be able to get it merged into 2.5 at some point...
Re:AFAICT Mosix is completely worthless (Score:2, Insightful)
FAQ from Website (Score:3, Informative)
Can I mix Mosix and openMosix nodes in the same cluster?
No. Just like the older Mosix, you should not mix nodes because the protocols are subject to un-announced changes from version to version. On top of that, every new version has bug fixes which warrant updating to the new kernels.
Whois the copyright holder of openMosix?
All the old Mosix code is copyright by Prof. Amnon Barak of Hebrew University of Jerusalem. All new code of openMosix is copyright 2002 by Moshe Bar, Tel Aviv.
How do I upgrade to openMosix?
openMosix maintains for now compatibility with the user-land tools of Mosix 1.5.2. I also have a port to openMosix of the user-land tools which will be released soon. To upgrade to openMosix, simply download the openMosix patch from www.openmosix.org and apply the patch with
patch -Np1 < openMosix1.5.2moshe
to a stock Linux kernel of 2.4.14 or 2.4.16 respectively. Make sure to get your old .config file (the .config file remains compatible) and recompile your kernel and modules. Then, reboot.
Is openMosix a fork of Mosix?
Right now, it is a pure fork. Eventually, it will become a separate clustering platform with its own distinct feature set and behaviour.
Some facts, anyone? (Score:2)
I really appreciate the work that Barak has done with and on Mosix. But I also find it kind of odd that Mosix could be the "property" of one individual. I would assume that it was developed with public research grants and while the author was employed at a university. Graduate students probably have also contributed, and there probably have been bug fixes as well. So, maybe it isn't bad if there is a GPL'ed distribution of Mosix after all. The GPL regulates issues of ownership rather well.
As for a user-space implementation of Mosix, I think that makes sense, although it has its drawbacks as well. One of the problems with user-space implementations is that they tend to be less than transparent in practice. It also strikes me as somewhat redundant, since Condor has already gone the user space route. A userland Mosix would only make sense if it were free and open source (as opposed to Condor).
Altogether, I hope people won't get too upset at each other over this. Mosix is great stuff and Barak and his university have been generous in making it available freely up to this point.
Re:Some facts, anyone? (Score:2)
I use MOSIX, love it... (Score:2)
MOSIX is great for general-purpose clustering. Processes are scattered across a cluster automatically without having to modify the programs. No API is needed other than usual Unix-level process use and it allows parallel execution of any program, although full use requires a parallel program design.
I just wish that it would go in the official linux kernel, something like
CONFIG_MOSIX=y
It's a great chance that Linux doesn't only play catch up with other flavors of Unix - it can take the leadership and give us the ability to create clusters using the tools in the standard distribution!
Re:I use MOSIX, love it... (Score:1)
The 2.4.13 patch even must have an effect on some SCSI code, because I can't compile a certain module for SCSI after MOSIX patching (one I didn't need anyway).
I seriously doubt we will ever see MOSIX in the main kernel tree, sorry.
Good to see. (Score:3, Interesting)
Anyway, I had noticed that 2.4.13 was sticking around awhile.. meanwhile, the page was being updated.. adding some information about a user-space version and the 'disclaimer' on the download page.
Mosix sounds like a good thing, but in reality it isn't very suitable for many of my common tasks. The biggest problem is the lack of support for programs using shared memory. Apache, Mysql, these do not migrate. Also, programs using Xwindows will not migrate well.. as they are network bound and will migrate back to their home node once they need to report the the Xserver. Basically, don't install mosix on 10 of your home machines and expect any kind of performance increase.. besides, you could probably toss out your dual 1ghz PIII, replace it with a 400mhz celeron and not notice a difference.
It would be nice if there was a transparent pseudo layer for things like X.. where all of the machines have their IP for communicating via mosix.. but all 'outside' communication would be made via mosix though a 'public' ip. That public IP would then be used to connect to the Xserver; hence, apps network bound could migrate easily as they would still have the same IP and (spoofed) Mac address. Basically it would be building a NAT router into Mosix for the idea of being able to migrate network-bound applications. It sounds more complicated then it is, but less complicated then it is to impliment.. and I probably don't make any sense, but I know what I mean
Re:No need for NAT router for open socket migratio (Score:1)
i think Mosix tries to get a cluster-wide process id, so if it's 16bit long, it could be in the class-B form [net.net.pid.pid], instead of [net.net.host.host]
when a process is migrated, it would send a broadcast message to change ARP tables, so the same IP address would be sent to a different MAC address
it would be better than setting every ethernet port in promiscous mode, since that woudn't work in a switched network (or in myrinet, i think)
Re:No need for NAT router for open socket migratio (Score:1)
Think of the effect on switched networks. Switches have finite-sized ARP caches, so by artificially inflating them you increase the number of misses (where each miss equates to a broadcast since the switch doesn't know where to send the packet).
Not a serious problem, if your number of hosts (on your network, not just in your cluster) << number of MAC addresses supported by your switches (a few thousand, often). I suppose it does depend on how many "away-from-home" processes start per unit time.
Also, each ARP packet will itself be broadcast, causing more traffic up front. Again, this may or may not be significant. And each host has to process and cache the ARP.
If you are assuming that the MOSIX cluster will be on a dedicated network - probably reasonable - none of this is much of a problem.
Re:No need for NAT router for open socket migratio (Score:1)
no, switches doesn't have ARP caches, they have MAC-Port tables. the ARP tables reside on the hosts.
the ARP-update packets (yet to be invented, i think) would include the new MAC address (or a special, host-bound IP address), that would be broadcast (bad, but only one packet), there wouldn't be a new ARP query.
in any case it woulbe FAR less traffic than the current moel of sending everything to the original node, just to be retransmitted. since in most scenarios every process would originate at the same node, it would easily saturate one leg of the switch.
a process-bound IP would allow network migration with just an extra broadcast packet for each migration (usually the migration needs to transfer a few megs per se)
Hey, check this out... (Score:2, Interesting)
MOSIX/Beowulf cluster troll. (Score:2)
Hey Moshe! Its OpenMosix + XFS (Score:2)
Coding wise the conflicts seemed trivial (and many times redundant). To minimize potential conflicts we don't use MFS and we don't use the debugger.
The only difficulty came when we started using Mosix 1.5.2, we had some issues where we get intermittent periods of "Too many open files" when a node goes down. Somehow we've avoided them for the past month, we think this may have more to do with AutoFS.
I'm wondering if you would like the diff from this? I'm also interested in helping with the DSM development and socket migration. I may be slow on the uptake but where can I start and help out?
Aptly named (Score:2, Insightful)
The word "bar" is Hebrew for "free"... Free as in speech, not beer, believe it or not!
Re:Aptly named (Score:1)
"son of" (which is not the case in my name)
or
"free, wild" (which by pure coincidence is the meaning of my name)
Moshe Bar
does it run linux? (Score:1)
QED
OpenMOSIX (Score:1)
wrong impression about this project.
For starters, MOSIX is MUCH easier to install than both PVM and MPI. In a
nutshell:
Step 1: Install userland programs
Step 2: Recompile kernel w/ MOSIX patch
Step 3: Setup LILO and reboot
Step 4: Repeat steps 2 & 3 for other nodes
NOTE: In my case, I only installed userland programs on the first node.
Although MOSIX is peer-to-peer, which is good, I'm using it with a master node.
Eventually, I plan to set up a second master node for redundancy.
NOTE #2: Most of the install is taken care of by using the install script.
As an example, it took me half a day to set up my first eight-node MOSIX
cluster, and 90% of that time was spent on the first node.
On User-MOSIX vs. Kernel-MOSIX: I think we may be missing an important point
here, as well:
It may be possible to incorporate U-MOSIX in places where K-MOSIX is not
possible (access to source code is required with K-MOSIX!). However, I do
think that a bit of salesmanship may be going into the description of U-MOSIX:
From the mosix.org website:
"U-MOSIX is better tuned for cluster and GRID computing, including the ability
to handle large number of short processes, run in heterogeneous clusters, with
different versions of Unix such as FreeBSD, Linux and Solaris"
How so? K-MOSIX is designed for hetero clustering. Large number of short
processes? The only way I can see this is with TCP/IP apps and socket
migration or with one helluva Myrinet connection between nodes. I can see the
advantage in *nix compatibility with userland MOSIX, but clustering
capabilities? I'll believe it when I see it. One of the principle advantages
of MOSIX that I have seen over Condor and the like is that MOSIX is
kernel-level. I've done my share of benchmarks, and nothing comes close.
The statements on the mosix.org website lead me to believe that U-MOSIX is
headed for commercial space. I hold no affiliation with either Professor
Barak or Moshe Bar, but I will say this: If Prof. Barak can produce
user-level code that outperforms K-MOSIX, then he deserves commercial
compensation and I will be the first in line to order.
I think that both gentlemen have done an outstanding job in producing a tool to
be used by researchers around the globe. My greatest hope is that this
development will spur competition (innovation is a better word, but WAY
overused!) between the two organizations.
Political Sidenote: I do feel that public educational funding should == open
source, but federal research funding is a different matter. This type of
funding is typically project-based, and IMHO the licensing of such should be
the decision of the government involved. I would, however, like to know the
specific problem that Professor Barak has with the GPL (assuming that the
rumors are true).