[R-pkg-devel] Creating a Makevars that includes objects from another compiler
Greetings, Quick context, I have been working on developing a set of packages to make GPU computing as simple as possible with R. I have a functional package that works very nicely with OpenCL based code (gpuR - https://github.com/cdeterman/gpuR) but I also am building a companion CUDA backend package (gpuRcuda - https://github.com/cdeterman/gpuRcuda) which is stripped down to a single function at the moment to work out the compiling issue described below. The problem is, CUDA requires me to use the NVCC compiler. So I cannot just rely on the magic of Rcpp, I need to create a Makevars file. I have done so and I am very close but I cannot get the shared object file to be created with NVCC. The individual files compile with NVCC just fine, the build script just keeps switching back to g++ for the shared object file. A quick excerpt from my build script ''' /usr/local/cuda-7.0/bin/nvcc -arch=sm_30 -Xcompiler -fPIC -Xcudafe --diag_suppress=boolean_controlling_expr_is_constant -I/usr/share/R/include -I/usr/include -I/home/cdeterman/R/x86_64-pc-linux-gnu-library/3.2/RViennaCL/include -I"/home/cdeterman/R/x86_64-pc-linux-gnu-library/3.2/RcppEigen/include" viennacl_sgemm.cu -c # NOW IT SWITCHES TO G++? ccache g++-4.8 -shared -L/usr/lib/R/lib -Wl,-Bsymbolic-functions -Wl,-z,relro -o gpuRcuda.so RcppExports.o viennacl_cudaMatrix_sgemm.o -L/usr/lib/R/lib -lR ''' You can see my current Makevars file here ( https://github.com/cdeterman/gpuRcuda/blob/master/src/Makevars). I have read in the 'Writing R Extensions' manual that this is possible with the OBJECTS macro ( http://cran.r-project.org/doc/manuals/r-release/R-exts.html#Creating-shared-objects) however I cannot figure out how to use it properly (and haven't been able to find an example). I have looked at the other packages on CRAN that also use CUDA (such as WideLM which I tried to emulate) but I can't seem to figure out why R keeps defaulting the shared library creation to g++. Perhaps my shared object section of my Makevars is incorrect? Any insights would be sincerely appreciated, Charles [[alternative HTML version deleted]] __ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel
Re: [R-pkg-devel] Creating a Makevars that includes objects from another compiler
A quick followup, even when I try and use the g++ I cannot seem to pass the objects (it keeps omitting the .o file generated from nvcc). How can I have my Makevars file pass a defined list of object files to the final shared library call? Alternate gpuRcuda.so block where the build output still doesn't use the OBJS variable (note, I confirmed that the OBJS variable does contain all .o files) gpuRcuda.so: $(OBJS) $(CXX) $< -o $@ $(R_LIBS) $(LIBS) Regards, Charles On Fri, Jun 5, 2015 at 9:42 AM, Charles Determan wrote: > Greetings, > > Quick context, I have been working on developing a set of packages to make > GPU computing as simple as possible with R. I have a functional package > that works very nicely with OpenCL based code (gpuR - > https://github.com/cdeterman/gpuR) but I also am building a companion > CUDA backend package (gpuRcuda - https://github.com/cdeterman/gpuRcuda) > which is stripped down to a single function at the moment to work out the > compiling issue described below. > > The problem is, CUDA requires me to use the NVCC compiler. So I cannot > just rely on the magic of Rcpp, I need to create a Makevars file. I have > done so and I am very close but I cannot get the shared object file to be > created with NVCC. The individual files compile with NVCC just fine, the > build script just keeps switching back to g++ for the shared object file. > > A quick excerpt from my build script > > ''' > /usr/local/cuda-7.0/bin/nvcc -arch=sm_30 -Xcompiler -fPIC -Xcudafe > --diag_suppress=boolean_controlling_expr_is_constant -I/usr/share/R/include > -I/usr/include > -I/home/cdeterman/R/x86_64-pc-linux-gnu-library/3.2/RViennaCL/include > -I"/home/cdeterman/R/x86_64-pc-linux-gnu-library/3.2/RcppEigen/include" > viennacl_sgemm.cu -c > > # NOW IT SWITCHES TO G++? > > ccache g++-4.8 -shared -L/usr/lib/R/lib -Wl,-Bsymbolic-functions > -Wl,-z,relro -o gpuRcuda.so RcppExports.o viennacl_cudaMatrix_sgemm.o > -L/usr/lib/R/lib -lR > ''' > > You can see my current Makevars file here ( > https://github.com/cdeterman/gpuRcuda/blob/master/src/Makevars). > > I have read in the 'Writing R Extensions' manual that this is possible > with the OBJECTS macro ( > http://cran.r-project.org/doc/manuals/r-release/R-exts.html#Creating-shared-objects) > however I cannot figure out how to use it properly (and haven't been able > to find an example). I have looked at the other packages on CRAN that also > use CUDA (such as WideLM which I tried to emulate) but I can't seem to > figure out why R keeps defaulting the shared library creation to g++. > Perhaps my shared object section of my Makevars is incorrect? > > Any insights would be sincerely appreciated, > Charles > > [[alternative HTML version deleted]] __ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel
Re: [R-pkg-devel] Creating a Makevars that includes objects from another compiler
I ultimately found more details in the 'Writing R Extensions' in another section (http://www.hep.by/gnu/r-patched/r-exts/R-exts_11.html). I needed to assign the object files to the OBJECTS variable with an '=' sign and remove anything below the .so block. The new block that successfully compiled looked like this: OBJECTS = $(cu_sharedlibs) $(cpp_sharedlibs) ... gpuRcuda.so: $(OBJECTS) By using OBJECTS instead of OBJS it overwrote R's default assignment of object files to link. Cheers, Charles On Fri, Jun 5, 2015 at 11:13 AM, Neal Fultz wrote: > Weird. Maybe it's not using that target at all, and is instead using a > generic .so target? > > On Fri, Jun 5, 2015 at 9:02 AM, Charles Determan > wrote: > >> A quick followup, even when I try and use the g++ I cannot seem to pass >> the >> objects (it keeps omitting the .o file generated from nvcc). How can I >> have my Makevars file pass a defined list of object files to the final >> shared library call? >> >> Alternate gpuRcuda.so block where the build output still doesn't use the >> OBJS variable (note, I confirmed that the OBJS variable does contain all >> .o >> files) >> >> gpuRcuda.so: $(OBJS) >> $(CXX) $< -o $@ $(R_LIBS) $(LIBS) >> >> Regards, >> Charles >> >> On Fri, Jun 5, 2015 at 9:42 AM, Charles Determan >> wrote: >> >> > Greetings, >> > >> > Quick context, I have been working on developing a set of packages to >> make >> > GPU computing as simple as possible with R. I have a functional package >> > that works very nicely with OpenCL based code (gpuR - >> > https://github.com/cdeterman/gpuR) but I also am building a companion >> > CUDA backend package (gpuRcuda - https://github.com/cdeterman/gpuRcuda) >> > which is stripped down to a single function at the moment to work out >> the >> > compiling issue described below. >> > >> > The problem is, CUDA requires me to use the NVCC compiler. So I cannot >> > just rely on the magic of Rcpp, I need to create a Makevars file. I >> have >> > done so and I am very close but I cannot get the shared object file to >> be >> > created with NVCC. The individual files compile with NVCC just fine, >> the >> > build script just keeps switching back to g++ for the shared object >> file. >> > >> > A quick excerpt from my build script >> > >> > ''' >> > /usr/local/cuda-7.0/bin/nvcc -arch=sm_30 -Xcompiler -fPIC -Xcudafe >> > --diag_suppress=boolean_controlling_expr_is_constant >> -I/usr/share/R/include >> > -I/usr/include >> > -I/home/cdeterman/R/x86_64-pc-linux-gnu-library/3.2/RViennaCL/include >> > -I"/home/cdeterman/R/x86_64-pc-linux-gnu-library/3.2/RcppEigen/include" >> > viennacl_sgemm.cu -c >> > >> > # NOW IT SWITCHES TO G++? >> > >> > ccache g++-4.8 -shared -L/usr/lib/R/lib -Wl,-Bsymbolic-functions >> > -Wl,-z,relro -o gpuRcuda.so RcppExports.o viennacl_cudaMatrix_sgemm.o >> > -L/usr/lib/R/lib -lR >> > ''' >> > >> > You can see my current Makevars file here ( >> > https://github.com/cdeterman/gpuRcuda/blob/master/src/Makevars). >> > >> > I have read in the 'Writing R Extensions' manual that this is possible >> > with the OBJECTS macro ( >> > >> http://cran.r-project.org/doc/manuals/r-release/R-exts.html#Creating-shared-objects >> ) >> > however I cannot figure out how to use it properly (and haven't been >> able >> > to find an example). I have looked at the other packages on CRAN that >> also >> > use CUDA (such as WideLM which I tried to emulate) but I can't seem to >> > figure out why R keeps defaulting the shared library creation to g++. >> > Perhaps my shared object section of my Makevars is incorrect? >> > >> > Any insights would be sincerely appreciated, >> > Charles >> > >> > >> >> [[alternative HTML version deleted]] >> >> __ >> R-package-devel@r-project.org mailing list >> https://stat.ethz.ch/mailman/listinfo/r-package-devel >> > > [[alternative HTML version deleted]] __ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel
Re: [R-pkg-devel] Managing RNG in C code
The answer is do not try to manage the RNG from C. Just do GetRNGstate(); then generate a lot of random variates and then PutRNGstate(); just before leaving. There is only one call to each in one call of C from R. This is TRT (the Right Thing). Anything else is TWT (the Wrong Thing). Starting an RNG a whole bunch of times with lots of different seeds doesn't do random number generation. You want one continuous stream. Only then does the RNG have the properties claimed for it. Moreover, you want your code to follow the R way (the tao of R). Your use of RNG should be just like every other R function. The user should be able to save .Random.seed and restore it and get the same results with another run. So the user is doing reproducible research. The user shouild be able to use RNGkind to change the random number generator. If you muck about with this you get in the way of the users. Your code is less useful not more. In my packages that use random number generation (especially mcmc), this is the policy I follow. The only thing I do with the random number generator is save .Random.seed before going from R to C and store the saved value in the returned object so the user does not have to. As with many problems, the right answer is don't do that! Message: 1 > Date: Thu, 4 Jun 2015 19:47:27 +0200 > From: Guillaume Chapron > To: r-package-devel@r-project.org > Subject: [R-pkg-devel] Managing RNG in C code > Message-ID: > Content-Type: text/plain; charset=us-ascii > > Hello, > > I am working on a package where I am passing some computations from R to C > and back. The C code runs many stochastic trajectories of a population > model (with the RNG from R through multiple calls to GetRNGstate and > PutRNGstate). I would like that each trajectory has its own seed. > > My problem is that the guidelines to develop R packages write in the > section '6.3 Random number generation' that "The random number generator is > private to R; there is no way to select the kind of RNG or set the seed > except by evaluating calls to the R functions." > > The only way I see to get around this is to set the seed in R, run 1 > trajectory in C, get results back in R, increment the seed in R, run a 2nd > trajectory in C with this new seed, add the new results to the previous one > in R, and repeat this e.g. 1000 times. Does this sound fine or should I be > concerned that calling the C library so many times may slow down the > computation (even if the library is only loaded 1 time at the beginning), > e.g. is there a substantial overhead in interfacing from R to C and back? > > Alternatively, is there a hack to get access by force to the private RNG > in R from C and set its seed? This way, I could call the C code only 1 time > and run the 1000 trajectories within C, updating the seed in R for each new > trajectory (this is what I did when I previously used the GSL). > > Thanks for any advice on this! > > Guillaume > -- Charles Geyer Professor, School of Statistics University of Minnesota char...@stat.umn.edu [[alternative HTML version deleted]] __ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel
[R-pkg-devel] Keeping secrets in R packages
I am working on updating the RGoogleDocs package to use OAuth2, since the old api was recently turned off (at my job we use it as a poor mans database). As part of the auth process, I need to send google a client key and secret in order to gain access to a spreadsheet. Is there a 'safe' place to store the secret? I would strongly prefer not requiring my end users to have to sign up for their own google API keys, but I also do not want to store the secret in the clear. Any suggestions on best practices? -Neal Fultz [[alternative HTML version deleted]] __ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel
Re: [R-pkg-devel] Keeping secrets in R packages
On 05/06/2015 6:19 PM, Neal Fultz wrote: > I am working on updating the RGoogleDocs package to use OAuth2, since the > old api was recently turned off (at my job we use it as a poor mans > database). > > As part of the auth process, I need to send google a client key and secret > in order to gain access to a spreadsheet. Is there a 'safe' place to store > the secret? > I would strongly prefer not requiring my end users to have to sign up for > their own google API keys, but I also do not want to store the secret in > the clear. > > Any suggestions on best practices? I don't think there is any way to do this. You could tell people that you will email the key on request, but there's no way to tell them something without telling them. Duncan Murdoch __ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel
Re: [R-pkg-devel] Keeping secrets in R packages
On 5 June 2015 at 15:19, Neal Fultz wrote: | I am working on updating the RGoogleDocs package to use OAuth2, since the | old api was recently turned off (at my job we use it as a poor mans | database). | | As part of the auth process, I need to send google a client key and secret | in order to gain access to a spreadsheet. Is there a 'safe' place to store | the secret? | I would strongly prefer not requiring my end users to have to sign up for | their own google API keys, but I also do not want to store the secret in | the clear. | | Any suggestions on best practices? I am not sure what you can call 'best' but what is getting common for myself is to source a file below $HOME, ie ~/.somethingrc, on launch. See for example the CRAN package RPushBullet which also uses an OAuth2 interface: https://github.com/eddelbuettel/rpushbullet/blob/master/R/init.R#L21-L38 It stores the (JSON-encoded, JSON was already a given in the context) object in a per-package environment, and for ease of other accessors also stores them in the global options(). Dirk -- http://dirk.eddelbuettel.com | @eddelbuettel | e...@debian.org __ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel
Re: [R-pkg-devel] Keeping secrets in R packages
Have you checked out Hadley's secret package? https://github.com/hadley/secure/blob/master/README.md Sent from my iPhone On 6 Jun 2015, at 8:19 am, Neal Fultz wrote: I am working on updating the RGoogleDocs package to use OAuth2, since the old api was recently turned off (at my job we use it as a poor mans database). As part of the auth process, I need to send google a client key and secret in order to gain access to a spreadsheet. Is there a 'safe' place to store the secret? I would strongly prefer not requiring my end users to have to sign up for their own google API keys, but I also do not want to store the secret in the clear. Any suggestions on best practices? -Neal Fultz [[alternative HTML version deleted]] __ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel [[alternative HTML version deleted]] __ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel
