Re: [Rd] Embedded R Fails When Run on LSF Queue System
Thanks Dirk. I'll confess I am mostly trying to follow the practices
outlined in the writing R-extensions manual and books by Chambers and
Gentleman, but have found some conflicting advice on the internet in
various places, and am also trying to build off a previously existing code
base (that at least worked on some
operating-systems/architectures/scenarios, but seems to break when I move
the program around to any new type of environment). Will definitely
re-write that bit.
Sincere thanks again. Out of curiosity, can I ask one more question since I
gather you have much more expertise than myself? The extension manual has a
somewhat obscure, to me at least, section on threading issues (8.1.5). My
understanding is that because the R_CStackStart is a single value, if R is
called from multiple threads (even if never simultaneously), it is
essentially best to just disable the stack check rather than try to re-write
this value each time, which I gather might need to be done (and simply
increasing the stack size of the calling thread will not help). Is this the
approach you have taken in your projects? Or do you think this is also a
bad idea?
-Original Message-
From: Dirk Eddelbuettel [mailto:e...@debian.org]
Sent: Sunday, November 10, 2013 8:50 PM
To: Nigel Delaney
Cc: r-devel@r-project.org
Subject: Re: [Rd] Embedded R Fails When Run on LSF Queue System
On 10 November 2013 at 16:32, Nigel Delaney wrote:
| I had a quick question I was hoping someone might be able to answer, as my
journey through the source code so far has not been very profitable. I have
a command line program that embeds R, and the program works just fine when
run from the command line. However, when I run the program as a job in the
LSF cluster at my institute, the following error occurs:
|
| Fatal error: you must specify '--save', '--no-save' or '--vanilla'
|
| This is confusing becasue I believe I have set this by changing the
startup parameters so that the no-save option is in use. I initialize R with
the following collection of function calls:
|
| 1-Rf_initialize_R
| 2-R_SetParams
| 3-setup_Rmainloop
|
| At step 2 I pass in a structure that specifies the no-save option (as well
as interactive=true, verbose=false), so I would expect it to be okay, but I
still receive this error. Again, not at the terminal but only when run as a
separate process.
|
| I gathered from this thread that someone else has experienced this
(https://stat.ethz.ch/pipermail/r-help/2008-March/155935.html), but am not
sure how best to rectify it as I believe I have already changed the setting.
|
| Does anyone know what a good solution or work around might be?
As a first approximation, when the manual tells you do something a certain
way, you are in fact better off doing it that way.
I stand behind two projects embedding R: littler (with Jeff Horner), and
RInside (with Romain Francois). Both do this explicitly. From littler.c:
char *R_argv[] = {(char*)programName, "--gui=none", "--no-restore",
"--no-save", "--no-readline", "--silent", "", ""};
char *R_argv_opt[] = {"--vanilla", "--slave"};
int R_argc = (sizeof(R_argv) - sizeof(R_argv_opt) ) /
sizeof(R_argv[0]);
[...]
/* some logic to add R_argv_opt parts to R_argv omitted (/
[...]
Rf_initEmbeddedR(R_argc, R_argv); /* Initialize the embedded R
interpreter */
Hope this helps, Dirk
--
Dirk Eddelbuettel | e...@debian.org | http://dirk.eddelbuettel.com
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
[Rd] SUGGESTION: Environment variable R_MAX_MC_CORES for maximum number of cores
I like to propose a unified/standard system environment variable that
specifies the maximum number of cores an R session should use, e.g.
R_MAX_MC_CORES. This could then be used to *guide* multicore
implementations on the number of cores to use. This is different from
parallel::detectCores().
ENVIRONMENT VARIABLE:
library(parallel)
mc.cores <- as.integer(Sys.getenv("R_MAX_MC_CORES", 1L))
res <- mclapply(1:10, FUN=fib, mc.cores=mc.cores)
R OPTION:
Analogously to several other env.var./options, R_MAX_MC_CORES could
set an option on startup for convenience, e.g.
options(max.mc.cores=as.integer(Sys.getenv("R_MAX_MC_CORES", 1L)))
R COMMAND-LINE OPTION:
One could also imagine a command-line option for R/Rscript that sets this, e.g.
Rscript --max.mc.cores=3 batch.R
EXAMPLE OF USAGE:
This would for instance simplify multicore processing on PBS cluster,
where the PBS job script can be:
Rscript --max.mc.cores=$PBS_NUM_PPN batch.R
such that R and the 'batch.R' script does not have to be aware of
settings/variables specific to PBS (or whatever cluster system is
used).
Finally, getOption("max.mc.cores", 1L) could possibly also be the new
default for the 'mc.cores' argument in 'parallel' functions.
/Henrik
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
[Rd] problem using rJava with parallel::mclapply
Dear all,
I got an issue trying to parse excel files in parallel using XLConnect, the
process hangs forever.
Martin Studer, the maintainer of XLConnect kindly investigated the issue,
identified rJava as a possible cause of the problem:
This does not work (hangs):
library(parallel)
require(rJava)
.jinit()
res <- mclapply(1:2, function(i) {
J("java.lang.Runtime")$getRuntime()$gc()
1
}, mc.cores = 2)
but this works:
library(parallel)
res <- mclapply(1:2, function(i) {
require(rJava)
.jinit()
J("java.lang.Runtime")$getRuntime()$gc()
1
}, mc.cores = 2)
To cite Martin, it seems to work with mclapply when the JVM process is
initialized after forking.
Is this a bug or a limitation of rJava ?
Or is there a good practice for rJava clients to avoid this problem ?
Best,
Karl
P.S.
> sessionInfo()
R version 3.0.1 (2013-05-16)
Platform: x86_64-unknown-linux-gnu (64-bit)
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=C LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
loaded via a namespace (and not attached):
[1] tools_3.0.1
[[alternative HTML version deleted]]
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] problem using rJava with parallel::mclapply
Karl,
I have the following notes to self that may be pertinent:
options(java.parameters=
## Must preceed `library(XLConnect)` in order to prevent "Java
## requested System.exit(130), closing R." which happens when
## rJava quits R upon trapping INT (control-c), as is done by
## XLConnect (and playwith?), below. (c.f.:
## https://www.rforge.net/bugzilla/show_bug.cgi?id=237)
"-Xrs")
~Malcolm
>-Original Message-
>From: r-devel-boun...@r-project.org [mailto:r-devel-boun...@r-project.org] On
>Behalf Of Karl Forner
>Sent: Monday, November 11, 2013 11:41 AM
>To: r-devel@r-project.org
>Cc: Martin Studer
>Subject: [Rd] problem using rJava with parallel::mclapply
>
>Dear all,
>
>I got an issue trying to parse excel files in parallel using XLConnect, the
>process hangs forever.
>Martin Studer, the maintainer of XLConnect kindly investigated the issue,
>identified rJava as a possible cause of the problem:
>
>This does not work (hangs):
>library(parallel)
>require(rJava)
>.jinit()
>res <- mclapply(1:2, function(i) {
> J("java.lang.Runtime")$getRuntime()$gc()
> 1
> }, mc.cores = 2)
>
>but this works:
>library(parallel)
>res <- mclapply(1:2, function(i) {
> require(rJava)
> .jinit()
> J("java.lang.Runtime")$getRuntime()$gc()
> 1
>}, mc.cores = 2)
>
>To cite Martin, it seems to work with mclapply when the JVM process is
>initialized after forking.
>
>Is this a bug or a limitation of rJava ?
>Or is there a good practice for rJava clients to avoid this problem ?
>
>Best,
>Karl
>
>P.S.
>> sessionInfo()
>R version 3.0.1 (2013-05-16)
>Platform: x86_64-unknown-linux-gnu (64-bit)
>
>locale:
> [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
> [3] LC_TIME=en_US.UTF-8LC_COLLATE=en_US.UTF-8
> [5] LC_MONETARY=en_US.UTF-8LC_MESSAGES=en_US.UTF-8
> [7] LC_PAPER=C LC_NAME=C
> [9] LC_ADDRESS=C LC_TELEPHONE=C
>[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
>
>attached base packages:
>[1] stats graphics grDevices utils datasets methods base
>
>loaded via a namespace (and not attached):
>[1] tools_3.0.1
>
> [[alternative HTML version deleted]]
>
>__
>R-devel@r-project.org mailing list
>https://stat.ethz.ch/mailman/listinfo/r-devel
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] problem using rJava with parallel::mclapply
Thanks Malcolm,
But it does seem to solve the problem.
On Mon, Nov 11, 2013 at 6:48 PM, Cook, Malcolm wrote:
> Karl,
>
> I have the following notes to self that may be pertinent:
>
> options(java.parameters=
> ## Must preceed `library(XLConnect)` in order to prevent "Java
> ## requested System.exit(130), closing R." which happens when
> ## rJava quits R upon trapping INT (control-c), as is done by
> ## XLConnect (and playwith?), below. (c.f.:
> ## https://www.rforge.net/bugzilla/show_bug.cgi?id=237)
> "-Xrs")
>
>
> ~Malcolm
>
>
>
> >-Original Message-
> >From: r-devel-boun...@r-project.org [mailto:
> r-devel-boun...@r-project.org] On Behalf Of Karl Forner
> >Sent: Monday, November 11, 2013 11:41 AM
> >To: r-devel@r-project.org
> >Cc: Martin Studer
> >Subject: [Rd] problem using rJava with parallel::mclapply
> >
> >Dear all,
> >
> >I got an issue trying to parse excel files in parallel using XLConnect,
> the
> >process hangs forever.
> >Martin Studer, the maintainer of XLConnect kindly investigated the issue,
> >identified rJava as a possible cause of the problem:
> >
> >This does not work (hangs):
> >library(parallel)
> >require(rJava)
> >.jinit()
> >res <- mclapply(1:2, function(i) {
> > J("java.lang.Runtime")$getRuntime()$gc()
> > 1
> > }, mc.cores = 2)
> >
> >but this works:
> >library(parallel)
> >res <- mclapply(1:2, function(i) {
> > require(rJava)
> > .jinit()
> > J("java.lang.Runtime")$getRuntime()$gc()
> > 1
> >}, mc.cores = 2)
> >
> >To cite Martin, it seems to work with mclapply when the JVM process is
> >initialized after forking.
> >
> >Is this a bug or a limitation of rJava ?
> >Or is there a good practice for rJava clients to avoid this problem ?
> >
> >Best,
> >Karl
> >
> >P.S.
> >> sessionInfo()
> >R version 3.0.1 (2013-05-16)
> >Platform: x86_64-unknown-linux-gnu (64-bit)
> >
> >locale:
> > [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
> > [3] LC_TIME=en_US.UTF-8LC_COLLATE=en_US.UTF-8
> > [5] LC_MONETARY=en_US.UTF-8LC_MESSAGES=en_US.UTF-8
> > [7] LC_PAPER=C LC_NAME=C
> > [9] LC_ADDRESS=C LC_TELEPHONE=C
> >[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
> >
> >attached base packages:
> >[1] stats graphics grDevices utils datasets methods base
> >
> >loaded via a namespace (and not attached):
> >[1] tools_3.0.1
> >
> > [[alternative HTML version deleted]]
> >
> >__
> >R-devel@r-project.org mailing list
> >https://stat.ethz.ch/mailman/listinfo/r-devel
>
[[alternative HTML version deleted]]
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] problem using rJava with parallel::mclapply
On Nov 11, 2013, at 12:40 PM, Karl Forner wrote:
> Dear all,
>
> I got an issue trying to parse excel files in parallel using XLConnect, the
> process hangs forever.
> Martin Studer, the maintainer of XLConnect kindly investigated the issue,
> identified rJava as a possible cause of the problem:
>
> This does not work (hangs):
> library(parallel)
> require(rJava)
> .jinit()
> res <- mclapply(1:2, function(i) {
> J("java.lang.Runtime")$getRuntime()$gc()
> 1
> }, mc.cores = 2)
>
> but this works:
> library(parallel)
> res <- mclapply(1:2, function(i) {
> require(rJava)
> .jinit()
> J("java.lang.Runtime")$getRuntime()$gc()
> 1
> }, mc.cores = 2)
>
> To cite Martin, it seems to work with mclapply when the JVM process is
> initialized after forking.
>
> Is this a bug or a limitation of rJava ?
It’s a limitation of the Java runtime - you cannot fork a JVM. This is true for
most libraries that use long-lived threads or any kind if UI (see the warnings
in mcfork).
Cheers,
Simon
> Or is there a good practice for rJava clients to avoid this problem ?
>
> Best,
> Karl
>
> P.S.
>> sessionInfo()
> R version 3.0.1 (2013-05-16)
> Platform: x86_64-unknown-linux-gnu (64-bit)
>
> locale:
> [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
> [3] LC_TIME=en_US.UTF-8LC_COLLATE=en_US.UTF-8
> [5] LC_MONETARY=en_US.UTF-8LC_MESSAGES=en_US.UTF-8
> [7] LC_PAPER=C LC_NAME=C
> [9] LC_ADDRESS=C LC_TELEPHONE=C
> [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
>
> attached base packages:
> [1] stats graphics grDevices utils datasets methods base
>
> loaded via a namespace (and not attached):
> [1] tools_3.0.1
>
> [[alternative HTML version deleted]]
>
> __
> R-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
