Don't know the answer, partly because you have not shown enough of your code.
The reason small, reproducible examples are specified in the footer and Posting
Guide is to avoid this problem.
AFAIK all of the parallel processing libraries in R re-use the child processes,
so garbage collection could be an issue. The nested for loops are probably a
red herring, though, since that simply expands the number of iterations. The
total number of iterations is probably relevant, and the setup of your parallel
processes is probably important, as well as your sessionInfo().
---------------------------------------------------------------------------
Jeff Newmiller The ..... ..... Go Live...
DCN:<jdnew...@dcn.davis.ca.us> Basics: ##.#. ##.#. Live Go...
Live: OO#.. Dead: OO#.. Playing
Research Engineer (Solar/Batteries O.O#. #.O#. with
/Software/Embedded Controllers) .OO#. .OO#. rocks...1k
---------------------------------------------------------------------------
Sent from my phone. Please excuse my brevity.
On October 29, 2014 1:48:05 AM PDT, Alexander Engelhardt
<a...@chaotic-neutral.de> wrote:
>Hello all,
>
>I have a triple nested loop in R like this:
>
>all <- list()
>for(a in A){
> all[[a]] <- list()
> for(b in B){
> all[[a]][[b]] <- foreach(c=C, .combine=rbind) %dopar% {
> ## I'm leaving out some preprocessing here
> this_GAM <- gam(formula, data=data, family=nb(link="log",
>theta=THETA))
> predict(this_GAM, newdata=newdata)
> }
> }
>}
>
>The problem I have is that, with time, the individual R processes which
>
>the %dopar% spawns use up more and more RAM. When I start the triple
>loop, each process requires about 2GB of RAM, but after around eight
>hours, they use >4GB each. Here's the first two lines of a 'top'
>output:
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>
>20880 engelhar 20 0 7042m 4.0g 2436 R 59.2 6.4 14:30.15 R
>
>20878 engelhar 20 0 7042m 4.3g 2436 D 53.5 6.8 14:07.18 R
>
>
>I don't understand how this can happen. To my understanding, as soon as
>
>the foreach loop is done, i.e. as soon as a new 'b' is chosen from 'B'
>in the second loop, the individual parallel R processes should
>terminate
>and release the memory. There should not be an increase of memory
>consumption over time.
>
>Does anyone know what is going on and how I can avoid this behavior?
>
>Thanks in advance,
> Alex Engelhardt
>
>______________________________________________
>R-help@r-project.org mailing list
>https://stat.ethz.ch/mailman/listinfo/r-help
>PLEASE do read the posting guide
>http://www.R-project.org/posting-guide.html
>and provide commented, minimal, self-contained, reproducible code.
______________________________________________
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
