[R-pkg-devel] install.R running out of memory
Hi, I’m developing an ML library that has R bindings… when installing the library with R CMD INSTALL the R process is running out of memory (50G+ ram) when doing: ** byte-compile and prepare package for lazy loading any ideas how i could debug this part of code, to figure out what is actually happening and why is there a memory leak? cheers, viktor __ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel
Re: [R-pkg-devel] install.R running out of memory
Hi Viktor, On 2 November 2019 at 13:09, Viktor Gal wrote: | I’m developing an ML library that has R bindings… when installing the library with R CMD INSTALL the R process is running out of memory (50G+ ram) when doing: | ** byte-compile and prepare package for lazy loading | | any ideas how i could debug this part of code, to figure out what is actually happening and why is there a memory leak? Easiest for us to help if we can see code -- so if you have a public repo somewhere please the link. I suspect you have some sort of recursion or circular dependency somewhere. It would be very hard for R to run out of 50gb. But we cannot say more. So maybe triage. In a situation like this when a (supposedly complete) package draft of mine fails "top-down" I often re-validate the toolchain "bottom-up" with a minimal package. If that works, keep adding pieces step by step from the 'not-working large package' to the 'small working' package while continuously ensuring that it still builds. Hope this helps, Dirk -- http://dirk.eddelbuettel.com | @eddelbuettel | e...@debian.org __ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel
Re: [R-pkg-devel] install.R running out of memory
Hi Dirk, so the project is open source, you can reproduce the error yourself (but note it’ll take a long time to actually compile it). steps for reproducing: git clone https://github.com/shogun-toolbox/shogun.git cd shogun git checkout feature/shared_ptr mkdir build cd build cmake -DINTERFACE_R=ON .. make make install (it requires tons of dependencies… if you have docker you can docker pull shogun/shogun-dev and run things inside the container) the make install part runs the R CMD INSTALL so that’ll cause the problem. but i’ve just uploaded the generated R code that causes the problem here, note the script is 7Mb i.e. 175k LoC, so you better wget/curl it: http://maeth.com/shogun.R cheers, viktor > On 2 Nov 2019, at 13:52, Dirk Eddelbuettel wrote: > > > Hi Viktor, > > On 2 November 2019 at 13:09, Viktor Gal wrote: > | I’m developing an ML library that has R bindings… when installing the > library with R CMD INSTALL the R process is running out of memory (50G+ ram) > when doing: > | ** byte-compile and prepare package for lazy loading > | > | any ideas how i could debug this part of code, to figure out what is > actually happening and why is there a memory leak? > > Easiest for us to help if we can see code -- so if you have a public repo > somewhere please the link. > > I suspect you have some sort of recursion or circular dependency > somewhere. It would be very hard for R to run out of 50gb. But we cannot say > more. > > So maybe triage. In a situation like this when a (supposedly complete) > package draft of mine fails "top-down" I often re-validate the toolchain > "bottom-up" with a minimal package. If that works, keep adding pieces step by > step from the 'not-working large package' to the 'small working' package > while continuously ensuring that it still builds. > > Hope this helps, Dirk > > -- > http://dirk.eddelbuettel.com | @eddelbuettel | e...@debian.org __ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel
Re: [R-pkg-devel] install.R running out of memory
Hi Victor, On 2 November 2019 at 13:58, Viktor Gal wrote: | so the project is open source, you can reproduce the error yourself (but note it’ll take a long time to actually compile it). steps for reproducing: | git clone https://github.com/shogun-toolbox/shogun.git | cd shogun | git checkout feature/shared_ptr | mkdir build | cd build | cmake -DINTERFACE_R=ON .. | make | make install | | (it requires tons of dependencies… if you have docker you can docker pull shogun/shogun-dev and run things inside the container) | | the make install part runs the R CMD INSTALL so that’ll cause the problem. | | but i’ve just uploaded the generated R code that causes the problem here, note the script is 7Mb i.e. 175k LoC, so you better wget/curl it: | http://maeth.com/shogun.R Ok, I am of course aware of Shogun, and with a large codebases that may even have even has auto-generated interfaces (e.g. Swig, not sure if Shogun has, it's been a while since I looked at) then something like this can happen. No easier answer from me, and I won't have time to dig into this. Sorry. Dirk -- http://dirk.eddelbuettel.com | @eddelbuettel | e...@debian.org __ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel
Re: [R-pkg-devel] install.R running out of memory
Hi Dirk, no worries, thnx for the feedback! cheers, viktor > On 2 Nov 2019, at 13:58, Viktor Gal wrote: > > Hi Dirk, > > so the project is open source, you can reproduce the error yourself (but note > it’ll take a long time to actually compile it). steps for reproducing: > git clone https://github.com/shogun-toolbox/shogun.git > cd shogun > git checkout feature/shared_ptr > mkdir build > cd build > cmake -DINTERFACE_R=ON .. > make > make install > > (it requires tons of dependencies… if you have docker you can docker pull > shogun/shogun-dev and run things inside the container) > > the make install part runs the R CMD INSTALL so that’ll cause the problem. > > but i’ve just uploaded the generated R code that causes the problem here, > note the script is 7Mb i.e. 175k LoC, so you better wget/curl it: > http://maeth.com/shogun.R > > cheers, > viktor > >> On 2 Nov 2019, at 13:52, Dirk Eddelbuettel wrote: >> >> >> Hi Viktor, >> >> On 2 November 2019 at 13:09, Viktor Gal wrote: >> | I’m developing an ML library that has R bindings… when installing the >> library with R CMD INSTALL the R process is running out of memory (50G+ ram) >> when doing: >> | ** byte-compile and prepare package for lazy loading >> | >> | any ideas how i could debug this part of code, to figure out what is >> actually happening and why is there a memory leak? >> >> Easiest for us to help if we can see code -- so if you have a public repo >> somewhere please the link. >> >> I suspect you have some sort of recursion or circular dependency >> somewhere. It would be very hard for R to run out of 50gb. But we cannot say >> more. >> >> So maybe triage. In a situation like this when a (supposedly complete) >> package draft of mine fails "top-down" I often re-validate the toolchain >> "bottom-up" with a minimal package. If that works, keep adding pieces step by >> step from the 'not-working large package' to the 'small working' package >> while continuously ensuring that it still builds. >> >> Hope this helps, Dirk >> >> -- >> http://dirk.eddelbuettel.com | @eddelbuettel | e...@debian.org > > __ > R-package-devel@r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-package-devel __ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel
