Re: [Rd] makeCluster hangs
Also using R-devel might help - the forking support in parallel has been made more robust against race conditions, but the changes are probably too substantial to port to 3.4.x. If you find how to cause a race condition using parallel/forking in R-devel, a report would be greatly appreciated. Tomas On 02/11/2018 09:51 PM, T. Florian Jaeger wrote: Dear Henrik, thank you, for the quick reply. Bizarrely enough, the problem vanished when I woke the computer from sleep (I had previously replicated the problem after several restarts of both R and the MacOS). I will follow-up if I can again replicate the problem. Florian On 2/10/18 4:39 PM, Henrik Bengtsson wrote: A few quick comments: * You mention R --vanilla, but make sure to try with parallel::makeCluster(), so that you don't happen to pick up snow::makeCluster() if 'snow' is attached and ahead of parallel on the search() path. * Try creating a single background worker, i.e. parallel::makeCluster(1L). * Try with cl <- future::makeClusterPSOCK(1L, verbose = TRUE), which gives the same thing, but it also show you some details on what it does internally; that may give some clues where it stalls. /Henrik On Sat, Feb 10, 2018 at 12:11 PM, T. Florian Jaeger wrote: Hi all, I can't get the functionality of the package parallel to work. Specifically, makeCluster() hangs when I run it. I first noticed the problem when trying to run Rstan with multiple cores and the traced it back to the core package parallel. The following results in R hanging after the call to makeCluster. library(parallel) # Calculate the number of cores no_cores <- detectCores() - 1 # Initiate cluster cl <- makeCluster(no_cores) I'm running MacOS High Sierra 10.13.3 (17D47) on a MacbookPro 2017 laptop with 4 cores. platform x86_64-apple-darwin15.6.0 arch x86_64 os darwin15.6.0 system x86_64, darwin15.6.0 status major 3 minor 4.3 year 2017 month 11 day30 svn rev73796 language R version.string R version 3.4.3 (2017-11-30) nickname Kite-Eating Tree The problem replicates in R --vanilla I've spent hours googling for solutions but can't find any reports of this problem. Any help would be appreciated. Florian __ R-devel@r-project.org mailing list https://urldefense.proofpoint.com/v2/url?u=https-3A__stat.ethz.ch_mailman_listinfo_r-2Ddevel&d=DwIBaQ&c=kbmfwr1Yojg42sGEpaQh5ofMHBeTl9EI2eaqQZhHbOU&r=O6dqVFPEDpdoXY3wkv8u6o0LHKx4WbQ_itn0O87jj5s&m=R7DIWWqYTP2xarhrvKcymtN3XlAQ9vHLFDhPL6FxQ60&s=7F8Lez-XE8iC2JBU4JYsEtF3U0HObhMnCASud5xTgNM&e= . __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
[Rd] [parallel] fixes load balancing of parLapplyLB
Dear R-Devel List,
**TL;DR:** The function **parLapplyLB** of the parallel package has
[reportedly][1] (see also attached RRD output) not
been doing its job, i.e. not actually balancing the load. My colleague Dirk
Sarpe and I found the cause of the problem
and we also have a patch to fix it (attached). A similar fix has also been
provided [here][2].
[1]:
https://stackoverflow.com/questions/38230831/why-does-parlapplylb-not-actually-balance-load
[2]: https://bugs.r-project.org/bugzilla3/show_bug.cgi?id=16792
## The Call Chain
First, we traced the relevant R function calls through the code, beginning with
`parLapplyLB`:
1. **parLapplyLB:** clusterApply.R:177, calls **splitList**, then
**clusterApplyLB**
2. **splitList:** clusterApply.R:157
3. **clusterApplyLB:** clusterApply.R:87, calls **dynamicClusterApply**
4. **dynamicClusterApply:** clusterApply.R:39
## splitList
We used both our whiteboard and an R session to manually *run* a few examples.
We were using lists of 100 elements and 5
workers. First, lets take a look at **splitList**:
```r
> sapply(parallel:::splitList(1:100, 5), length)
[1] 20 20 20 20 20
> sapply(parallel:::splitList(1:97, 5), length)
[1] 20 19 19 19 20
> sapply(parallel:::splitList(1:97, 20), length)
[1] 5 5 5 5 4 5 5 5 5 5 4 5 5 5 5 4 5 5 5 5
```
As we can see in the examples, the work is distributed as equally as possible.
## dynamicClusterApply
**dynamicClusterApply** works this way (simplified):
1. it first gives a chunk to each worker
2. once a worker comes back with the result, it is given the next chunk
**This is the important part:** As long as there are **more** chunks than
workers, there will be load balancing. If
there are fewer chunks than workers, each worker will get **at most one chunk**
and there is **no** load balancing.
## parLapplyLB
This is how **parLapplyLB** splits the input list (with a bit of refactoring,
for readability):
```r
parLapplyLB <- function(cl = NULL, X, fun, ...)
{
cl <- defaultCluster(cl)
chunks <- splitList(X, length(cl))
do.call(c,
clusterApplyLB(cl, x = chunks, fun = lapply, fun, ...),
quote = TRUE)
}
```
For our examples, the chunks have these sizes:
```r
> sapply(parallel:::splitList(1:100, 5), length)
[1] 20 20 20 20 20
```
There we have it: 5 chunks. 5 workers. With this work distribution, there can't
possibly be any load balancing, because
each worker is given a single chunk and then it stops working because there are
no more chunks.
Instead, **parLapplyLB** should look like this (patch is attached):
```r
parLapplyLB <- function(cl = NULL, X, fun, ...)
{
cl <- defaultCluster(cl)
chunkSize <- max(length(cl), ceiling(length(X) / length(cl)))
chunks <- splitList(X, chunkSize)
do.call(c,
clusterApplyLB(cl, x = chunks, fun = lapply, fun, ...),
quote = TRUE)
}
```
Examples with a cluster of 5 workers:
```r
# length(cl) < length(X)
> sapply(parallel:::splitList(1:100, ceiling(100 / 5)), length)
[1] 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5
# length(cl) >= length(X)
> sapply(parallel:::splitList(1:4, 4), length)
[1] 1 1 1 1
# one worker idles here, but we can't do better than that
```
With this patch, the number of chunks is larger than the number of workers, if
possible at all, and then load balancing
should work.
Best Regards
--
Christian Krause
Scientific Computing Administration and Support
Phone: +49 341 97 33144
Email: christian.kra...@idiv.de
German Centre for Integrative Biodiversity Research (iDiv) Halle-Jena-Leipzig
Deutscher Platz 5e
04103 Leipzig
Germany
iDiv is a research centre of the DFG – Deutsche Forschungsgemeinschaft
iDiv ist eine zentrale Einrichtung der Universität Leipzig im Sinne des § 92
Abs. 1 SächsHSFG und wird zusammen mit der
Martin-Luther-Universität Halle-Wittenberg und der
Friedrich-Schiller-Universität Jena betrieben sowie in Kooperation
mit dem Helmholtz-Zentrum für Umweltforschung GmbH – UFZ. Beteiligte
Kooperationspartner sind die folgenden
außeruniversitären Forschungseinrichtungen: das Helmholtz-Zentrum für
Umweltforschung GmbH - UFZ, das
Max-Planck-Institut für Biogeochemie (MPI BGC), das Max-Planck-Institut für
chemische Ökologie (MPI CE), das
Max-Planck-Institut für evolutionäre Anthropologie (MPI EVA), das
Leibniz-Institut Deutsche Sammlung von Mikroorganismen
und Zellkulturen (DSMZ), das Leibniz-Institut für Pflanzenbiochemie (IPB), das
Leibniz-Institut für Pflanzengenetik und
Kulturpflanzenforschung (IPK) und das Leibniz-Institut Senckenberg Museum für
Naturkunde Görlitz (SMNG). USt-IdNr. DE
14151038
[Rd] Fix minor typo in error message from grDevices
Hi,
I fixed a minor typo in an error message from grDevices. Please see
attached for a patch to revision 74246.
Thanks,
John
Index: src/library/grDevices/src/colors.c
===
--- src/library/grDevices/src/colors.c (revision 74246)
+++ src/library/grDevices/src/colors.c (working copy)
@@ -1529,7 +1529,7 @@
int n = length(val), *ians = INTEGER(ans);
for (int i = 0; i < PaletteSize; i++) ians[i] = (int)Palette[i];
if (n) {
- if (TYPEOF(val) != INTSXP) error("requires INTSXP argment");
+ if (TYPEOF(val) != INTSXP) error("requires INTSXP argument");
if (n > MAX_PALETTE_SIZE)
error(_("maximum number of colors is %d"), MAX_PALETTE_SIZE);
for (int i = 0; i < n; i++) Palette[i] = (rcolor)INTEGER(val)[i];
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
[Rd] Setting the path to Rtools for package compilation on Windows
Hi all, I'm trying to set up the Windows Rtools toolset for building packages with compiled code. I installed for Windows R-3.4.3 from CRAN and installed Rtools-3.4 in a custom location M:\R\R-3.4.3 and M:\R\Rtools-3.4 Following the instructions, in shell, I set Path=M:\R\Rtools-3.4\bin;M:\R\Rtools-3.4\gcc-4.6.3\bin;M:\R\R-3.4.3\bin;... (the ... are other paths irrelevant for R/Rtools). When I run M:\Work\RLibs>R.exe CMD INSTALL --build WGCNA I get the following ouput: In R CMD INSTALL * installing to library 'M:/R/R-3.4.3/library' * installing *source* package 'WGCNA' ... ** libs *** arch - i386 c:/Rtools/mingw_32/bin/g++ -I"M:/R/R-3.4.3/include" -DNDEBUG -O2 -Wall -mtune=generic -c bucketApproxSort.cc -o bucketApproxSort.o c:/Rtools/mingw_32/bin/g++: not found make: *** [bucketApproxSort.o] Error 127 Warning: running command 'make -f "Makevars.win" -f "M:/R/R-3.4.3/etc/i386/Makeconf" -f "M:/R/R-3.4.3/share/make/winshli b.mk" SHLIB_LDFLAGS='$(SHLIB_CXXLDFLAGS)' SHLIB_LD='$(SHLIB_CXXLD)' SHLIB="WGCNA.dll" OBJECTS="bucketApproxSort.o corFun ctions-common.o corFunctions-unified.o networkFunctions.o pivot.o quantileC.o"' had status 2 ERROR: compilation failed for package 'WGCNA' * removing 'M:/R/R-3.4.3/library/WGCNA' * restoring previous 'M:/R/R-3.4.3/library/WGCNA' Apparently the install is looking for Rtools in c:\Rtools. I am a perpetual Windows newbie and would be really thankful for any pointers as to how to proceed. Peter __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] Setting the path to Rtools for package compilation on Windows
I can confirm the behaviour that you report. Usually I put Rtools in c:\programs\Rtools and modify the path accordingly. Recently (don't recall for how long) I have encountered the same problems as you have and I have resorted to moving Rtools to c:\Rtools I have no idea as how to proceed; perhaps it could be worth trying an older version of Rtools (though that may cause other problems). Regards Søren On Mon, 2018-02-12 at 22:45 -0800, Peter Langfelder wrote: > Hi all, > > I'm trying to set up the Windows Rtools toolset for building packages > with compiled code. I installed for Windows R-3.4.3 from CRAN and > installed Rtools-3.4 in a custom location M:\R\R-3.4.3 and > M:\R\Rtools-3.4 > > Following the instructions, in shell, I set > Path=M:\R\Rtools-3.4\bin;M:\R\Rtools-3.4\gcc-4.6.3\bin;M:\R\R- > 3.4.3\bin;... > (the ... are other paths irrelevant for R/Rtools). > > When I run > > M:\Work\RLibs>R.exe CMD INSTALL --build WGCNA > > I get the following ouput: > > In R CMD INSTALL > * installing to library 'M:/R/R-3.4.3/library' > * installing *source* package 'WGCNA' ... > ** libs > > *** arch - i386 > c:/Rtools/mingw_32/bin/g++ -I"M:/R/R-3.4.3/include" -DNDEBUG > -O2 -Wall -mtune=generic -c bucketApproxSort.cc > -o bucketApproxSort.o > c:/Rtools/mingw_32/bin/g++: not found > make: *** [bucketApproxSort.o] Error 127 > Warning: running command 'make -f "Makevars.win" -f > "M:/R/R-3.4.3/etc/i386/Makeconf" -f "M:/R/R-3.4.3/share/make/winshli > b.mk" SHLIB_LDFLAGS='$(SHLIB_CXXLDFLAGS)' SHLIB_LD='$(SHLIB_CXXLD)' > SHLIB="WGCNA.dll" OBJECTS="bucketApproxSort.o corFun > ctions-common.o corFunctions-unified.o networkFunctions.o pivot.o > quantileC.o"' had status 2 > ERROR: compilation failed for package 'WGCNA' > * removing 'M:/R/R-3.4.3/library/WGCNA' > * restoring previous 'M:/R/R-3.4.3/library/WGCNA' > > > Apparently the install is looking for Rtools in c:\Rtools. I am a > perpetual Windows newbie and would be really thankful for any > pointers > as to how to proceed. > > Peter > > __ > R-devel@r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-devel __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
