Thank you Simon for the detailed reply. That explains much more of what I was looking for from the R side.
Dirk, I'm sorry if I seem hung up on anything here but I am trying to understand the details. My reply about XPtr or XPtr on arma/Eigen was to confirm my understanding was correct, which it appears it was. I was not aware the RVector/RMatrix objects don't connect to R as I am just now familiarizing myself with the package, that explains more of my confusion. I will look at doing work within the compiled code as you have suggested. Regards, Charles On Thu, May 12, 2016 at 9:18 AM, Dirk Eddelbuettel <e...@debian.org> wrote: > > On 12 May 2016 at 13:11, Mark van der Loo wrote: > | Charles, > | > | 1. Perhaps this question is better directed at the R-help or > | R-pacakge-devel mailinglist. > | > | 2. It basically means that R itself can only evaluate one R expression at > | the time. > | > | The parallel package circumvents this by starting multiple R-sessions and > | dividing workload. > | > | Compiled code called by R (such as C++ code through RCpp or C-code > through > | base R's interface) can execute multi-threaded code for internal > purposes, > | using e.g. openMP. A limitation is that compiled code cannot call R's C > API > | from multiple threads (in many cases). For example, it is not thread-safe > | to create R-variables from multiple threads running in C. (R's variable > | administration is such that the order of (un)making them from compiled > code > | matters). > > Well put. > > | I am not very savvy on Rcpp or XPtr objects, but it appears that Dirk > | provided answers about that in your SO-question. > > Charles seems to hang himself up completely about a small detail, failing > to > see the forest for the trees. > > There are (many) working examples of parallel (compiled) code with R. All > of > them stress (and I simplify here) that can you touch R objects, or call > back > into R, for fear of any assignment or allocation triggering an R event. R > being single-threaded it cannot do this. > > My answer to this problem is to only use non-R data structures. That is > what > RcpParallel does in the actual parallel code portions in all examples -- > types RVector and RMatrix do NOT connect back to R. There are several > working > examples. That is also what the OpenMP examples at the Rcpp Gallery do. > > Charles seems to be replying 'but I use XPtr' or 'I use XPtr on arma::mat > or > Eigen::Matrixxd' and seems to forget that these are proxy objects to SEXPs. > XPtr just wrap the SEXP for external pointers; Arma's and Eigen's matrices > are performant via RcppArmadillo and RcppEigen because we use R memory via > proxies. All of that is 'too close to R' for comfort. > > So the short answer is: enter compiled code from R, set a mutex (either > conceptually or explicitly), _copy_ your data in to plain C++ data > structures > and go to town in parallel via OpenMP and other multithreaded approaches. > Then collect the result, release the mutex and move back up. > > I hope this help. > > Dirk > > | > | Best, > | Mark > | > | > | > | > | > | > | > | > | > | > | Op do 12 mei 2016 om 14:46 schreef Charles Determan < > cdeterma...@gmail.com>: > | > | > R Developers, > | > > | > Could someone help explain what it means that R is single threaded? I > am > | > trying to understand what is actually going on inside R when users > want to > | > parallelize code. For example, using mclapply or foreach (with some > | > backend) somehow allows users to benefit from multiple CPUs. > | > > | > Similarly there is the RcppParallel package for RMatrix/RVector > objects. > | > But none of these address the general XPtr objects in Rcpp. Some > readers > | > here may recognize my question on SO ( > | > > | > > http://stackoverflow.com/questions/37167479/rcpp-parallelize-functions-that-return-xptr > | > ) > | > where I was curious about parallel calls to C++/Rcpp functions that > return > | > XPtr objects. I am being a little more persistent here as this > limitation > | > provides a very hard stop on the development on one of my packages that > | > heavily uses XPtr objects. It's not meant to be a criticism or > intended to > | > be rude, I just want to fully understand. > | > > | > I am willing to accept that it may be impossible currently but I want > to at > | > least understand why it is impossible so I can explain to future users > why > | > parallel functionality is not available. Which just echos my original > | > question, what does it mean that R is single threaded? > | > > | > Kind Regards, > | > Charles > | > > | > [[alternative HTML version deleted]] > | > > | > ______________________________________________ > | > R-devel@r-project.org mailing list > | > https://stat.ethz.ch/mailman/listinfo/r-devel > | > > | > | [[alternative HTML version deleted]] > | > | ______________________________________________ > | R-devel@r-project.org mailing list > | https://stat.ethz.ch/mailman/listinfo/r-devel > > -- > http://dirk.eddelbuettel.com | @eddelbuettel | e...@debian.org > [[alternative HTML version deleted]] ______________________________________________ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
