The R.batch package does most of this. It's been several years, but I used it run batch jobs on multiple (30-40) machines with different OS:es but sharing the same file system. No communication needed between hosts. It worked like a charm and I could process 3-4 CPU years in a few weeks for some bootstrap simulations. It got exception handling and much more. See also r-help thread 'R.batch (Was: Re: [R] Calling R from R and specifying "wait until script is finished")' on May 22, 2005:
https://www.stat.math.ethz.ch/pipermail/r-help/2005-May/071981.html The installation is no longer as documented there, but instead you can grab it from R-forge: http://r-forge.r-project.org/R/?group_id=428 This very moment, the r-forge server seems to be down, so you can also install a copy via: source("http://www.braju.com/R/hbLite.R";); hbLite("R.batch"); Then try this: library("R.batch"); example("JobBatch"); See the FileProgressBar class in the R.utils package for reporting progress via the file size (0-100 bytes), which allows you to use ls -l (or ftp remotely) to check the progress of your jobs. Feel free to do whatever you want with it. /Henrik On Tue, Sep 15, 2009 at 5:01 PM, ivo welch <ivo_we...@brown.edu> wrote: > I am about to write a "cluster-lite" R solution for myself. I wanted to > know whether it already exists. If not, I will probably write up how I do > this, and I will make the code available. > > Background: we have various linux and OSX systems, which are networked, but > not set up as a cluster. I have no one here to set up a cluster, so I need > a "hack" that facilitates parallel programming on standard networked > machines. I have accounts on all the machines, ssh access (of course > password-less), and networked file directory access. > > what I am ultimately trying to accomplish is built around a "simple" > function, that my master program would invoke: > > master.R: > multisystem( c("R slv.R 1 20 file1.out", "R slv.R 21 40 file2.out", "ssh > anotherhost R slv.R 41 80 file3.out"), announce=300) > > multisystem() should submit all jobs simultaneously and continue only after > all are completed. it should also tell me every 300 seconds what jobs it is > still waiting for, and which have completed. > > with basically no logic in the cluster, my master and slv programs have to > make up for it. master.R must have the smarts to know where it can spawn > jobs and how big each job should be. slv.R must have the smarts to place > its outputs into the marked files on the networked file directory. master.R > needs the smarts to combine the outputs of all jobs, and to resubmit jobs > that did not complete successfully. again, the main reason for doing all of > this is to avoid setting up a cluster across OSX and linux system, and still > to make parallel processing across linux/osx as easy as possible. I don't > think it gets much simpler than this. > > now, I know how to write the multisystem() in perl, but not in R. so, if I > roll it myself, I will probably rely on a mixed R/perl system here. This is > not desirable, but it is the only way I know how to do this. if something > like multisystem() already exists in R native, please let me know and save > me from reinventing the wheel. if it does not, some perl/R combo for this > soon will. > > regards, > > /iaw > > > -- > Ivo Welch (ivo.we...@brown.edu, ivo.we...@gmail.com) > > [[alternative HTML version deleted]] > > ______________________________________________ > R-help@r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. > ______________________________________________ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
